2023-07-02T08:35:25.918 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-02T08:35:25.919 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} --owner scheduled_matan@teuthology --archive /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293 --name matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi --verbose -- /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293/orig.config.yaml 2023-07-02T08:35:25.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-02T08:35:26.029 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293 branch: wip-yingxin-msgr-multi-core-crimson-10 description: crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} email: null first_in_suite: false job_id: '7324293' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yingxin-msgr-multi-core-crimson-10 ceph: conf: client: debug ms: 1 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd pool default crimson: true osd: crimson seastar smp: 3 debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 6450d2709f998063da1a187d826fea699691b0b1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 6450d2709f998063da1a187d826fea699691b0b1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 00edceddba022d4d35c4fe2bf8ae4808b752654c owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b sha1: 6450d2709f998063da1a187d826fea699691b0b1 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 00edceddba022d4d35c4fe2bf8ae4808b752654c targets: smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOqS6e1wxD0IKfQGrjPbQM7M7V4/e4hjGoWI/J5FO15c51f7uUO96HBA/cGQVUVNUmlre9YDyklL2gyLumPXPMBNP7LF4W9ordzlTcNmfKk7eMOQzbpGdVrDmPmb4jSA72Bm2LTYvspP0U1c7eFXvjM4mRYjHjl5IsdCTbdTSFhtR+pYPw6pk/zOX6IuISt9HALOtvguv3x/5x3q1+C7mUFIZMeJUjEoBrY2VHmoPtiWdeqs6PplgEwPBKUqG0PaYhnvMMN2yFrieUj4oAfEiUQ/VXMB1sS9cBCWCA2DgX4FhcF4NMuE96EPPjorl6hx6J43GWGEQHKyMTP/ptmuuGVDOInrAvlgJE5BpNKrtarM1TDFfCJy/MqwTVT/1L8n3wBIug6G1yis0lgmYIoZrrR4xmkThpiqB1WGkidBgeh/Ye2Gwv8JjELSuk+c4UH1PUW49yVeqQ4bCj/aKj6l+3V/Ty9vQa5QivjsgEqUs1SWtMfOcgBGpyH6nxmutQVoE= smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj/UFFLKldYHPXfTfKh85+du+x3iJh4dlxPOVHYlmglEVLCsVpbRknPbArvu1MyXmisW5Lknb9KYL7Xi6h+EvZfYiLpDdD4CoJAkrb73ldcQ9OnH5tPac+mDM/D4AxS5zv2uiRXWiBqgCS2vwiUpdh0eZFnCtysnBpzAYU5tyjdjKdGWjCvD2i+5DfJs/ciqeDW3coTuRx2GjwSYEwGlQ3w3toVgzayP25ZnzYIrK9sOQD4ZbxBnoWbt9//GEEcQ9G7DEP3g7tfuQAjy655jwS0R6eedNDDTFqrBVqVvKX2hZvGl8icnnSmeUzi7srcIkH7Xmrl98Jn4CR39/iwlcJsEk20+IanaAMwwmnV6leKhaDOExYc2bsk9/ZTHuv5Ufi8gRQbkwLTZoVIrveYsgXCCwf4lScUiKnTqJ9QhVzf9ZbOMXL/HYChzi1PmaYKX7d4bRqkyICyEGChqYqTWyB/DIURFVgkkP69UCBo+YoyyoKpn710e+1I/YG5VOe9xE= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b53f236c1077f1d295baafb9e0ca9a4b447019d7 timestamp: 2023-07-02_08:25:06 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-02T08:35:26.029 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa; will attempt to use it 2023-07-02T08:35:26.030 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks 2023-07-02T08:35:26.030 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-02T08:35:26.032 INFO:teuthology.task.internal:Checking packages... 2023-07-02T08:35:26.062 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '6450d2709f998063da1a187d826fea699691b0b1' 2023-07-02T08:35:26.062 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-02T08:35:26.063 INFO:teuthology.packaging:ref: None 2023-07-02T08:35:26.063 INFO:teuthology.packaging:tag: None 2023-07-02T08:35:26.063 INFO:teuthology.packaging:branch: wip-yingxin-msgr-multi-core-crimson-10 2023-07-02T08:35:26.063 INFO:teuthology.packaging:sha1: 6450d2709f998063da1a187d826fea699691b0b1 2023-07-02T08:35:26.064 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&ref=wip-yingxin-msgr-multi-core-crimson-10 2023-07-02T08:35:26.288 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4632.g6450d270 2023-07-02T08:35:26.288 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-02T08:35:26.294 INFO:teuthology.task.internal:no buildpackages task found 2023-07-02T08:35:26.294 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-02T08:35:26.347 INFO:teuthology.task.internal:Saving configuration 2023-07-02T08:35:26.359 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-02T08:35:26.381 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-02T08:35:26.411 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 08:25:11.161871', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCcOSju7dR+E1gJTeTIxnmoPeMHzSVRuIVeaAjOH0nFU2KXJvWupRL7Ot9qxVehr0h4Cug1uIo4w9dz/ZzQltgI='} 2023-07-02T08:35:26.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 08:25:11.160107', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCH/tNdKkgi1tDOnLBwxOCviKMo5ZueZrsNJv8KYrjH/t7yFqPfWHbczsA1s+6svjoXiDotXs3q38CxlJZX2uUs='} 2023-07-02T08:35:26.437 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-02T08:35:26.446 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2023-07-02T08:35:26.447 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-07-02T08:35:26.447 INFO:teuthology.run_tasks:Running task console_log... 2023-07-02T08:35:26.552 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4a343b5040>, signals=[15]) 2023-07-02T08:35:26.554 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-02T08:35:26.560 INFO:teuthology.task.internal:Opening connections... 2023-07-02T08:35:26.561 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2023-07-02T08:35:26.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:35:26.633 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2023-07-02T08:35:26.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:35:26.705 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-02T08:35:26.711 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2023-07-02T08:35:26.734 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2023-07-02T08:35:26.735 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2023-07-02T08:35:26.796 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2023-07-02T08:35:26.796 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="8" 2023-07-02T08:35:26.796 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2023-07-02T08:35:26.796 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2023-07-02T08:35:26.797 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="8" 2023-07-02T08:35:26.797 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el8" 2023-07-02T08:35:26.797 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-02T08:35:26.797 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2023-07-02T08:35:26.797 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-02T08:35:26.797 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2023-07-02T08:35:26.797 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-02T08:35:26.798 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-02T08:35:26.798 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-02T08:35:26.798 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2023-07-02T08:35:26.823 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2023-07-02T08:35:26.846 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2023-07-02T08:35:26.846 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2023-07-02T08:35:26.906 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2023-07-02T08:35:26.906 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8" 2023-07-02T08:35:26.907 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2023-07-02T08:35:26.907 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2023-07-02T08:35:26.907 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8" 2023-07-02T08:35:26.907 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8" 2023-07-02T08:35:26.907 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-02T08:35:26.907 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2023-07-02T08:35:26.907 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-02T08:35:26.907 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2023-07-02T08:35:26.908 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-02T08:35:26.908 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-02T08:35:26.908 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-02T08:35:26.908 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2023-07-02T08:35:26.932 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-02T08:35:26.942 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-02T08:35:26.981 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-02T08:35:26.982 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2023-07-02T08:35:26.984 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2023-07-02T08:35:27.004 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-02T08:35:27.013 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-02T08:35:27.013 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2023-07-02T08:35:27.045 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2023-07-02T08:35:27.076 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-02T08:35:27.137 INFO:teuthology.run_tasks:Running task kernel... 2023-07-02T08:35:27.153 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-02T08:35:27.154 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-02T08:35:27.154 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-02T08:35:27.155 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-02T08:35:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T08:35:27.176 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2023-07-02T08:35:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T08:35:27.177 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2023-07-02T08:35:27.238 INFO:teuthology.orchestra.run.smithi103.stdout:4.18.0-489.el8.x86_64 2023-07-02T08:35:27.239 INFO:teuthology.task.kernel:Running kernel on smithi103: 4.18.0-489.el8.x86_64 2023-07-02T08:35:27.239 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2023-07-02T08:35:27.240 INFO:teuthology.orchestra.run.smithi164.stdout:4.18.0-489.el8.x86_64 2023-07-02T08:35:27.240 INFO:teuthology.task.kernel:Running kernel on smithi164: 4.18.0-489.el8.x86_64 2023-07-02T08:35:27.241 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2023-07-02T08:35:32.220 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - AppStream 13 MB/s | 30 MB 00:02 2023-07-02T08:35:32.263 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - AppStream 13 MB/s | 30 MB 00:02 2023-07-02T08:36:08.475 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - BaseOS 1.3 MB/s | 40 MB 00:30 2023-07-02T08:36:10.284 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - BaseOS 1.2 MB/s | 40 MB 00:32 2023-07-02T08:36:14.534 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - Extras 62 kB/s | 18 kB 00:00 2023-07-02T08:36:15.117 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.6 kB 00:00 2023-07-02T08:36:16.501 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2023-07-02T08:36:17.052 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.6 kB 00:00 2023-07-02T08:36:17.628 INFO:teuthology.orchestra.run.smithi103.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-07-02T08:36:20.686 INFO:teuthology.orchestra.run.smithi164.stdout:Copr repo for python3-asyncssh owned by ceph 1.0 kB/s | 3.5 kB 00:03 2023-07-02T08:36:27.580 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 2.3 MB/s | 16 MB 00:06 2023-07-02T08:36:28.220 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 1.5 MB/s | 16 MB 00:10 2023-07-02T08:36:31.377 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 352 kB/s | 24 kB 00:00 2023-07-02T08:36:32.011 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 356 kB/s | 24 kB 00:00 2023-07-02T08:36:34.551 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-07-02T08:36:34.551 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-07-02T08:36:34.791 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:36:34.792 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:36:34.792 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2023-07-02T08:36:34.793 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:36:34.793 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:36:34.793 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 4.18.0-499.el8 baseos 9.8 M 2023-07-02T08:36:34.793 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:36:34.794 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core x86_64 4.18.0-499.el8 baseos 43 M 2023-07-02T08:36:34.794 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules x86_64 4.18.0-499.el8 baseos 35 M 2023-07-02T08:36:34.794 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:36:34.795 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:36:34.795 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:36:34.795 INFO:teuthology.orchestra.run.smithi164.stdout:Install 3 Packages 2023-07-02T08:36:34.795 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:36:34.796 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 88 M 2023-07-02T08:36:34.796 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 95 M 2023-07-02T08:36:34.796 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:36:35.236 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-07-02T08:36:35.236 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-07-02T08:36:35.470 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:36:35.471 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:36:35.471 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2023-07-02T08:36:35.471 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:36:35.471 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:36:35.471 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 4.18.0-499.el8 baseos 9.8 M 2023-07-02T08:36:35.472 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:36:35.472 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 4.18.0-499.el8 baseos 43 M 2023-07-02T08:36:35.472 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 4.18.0-499.el8 baseos 35 M 2023-07-02T08:36:35.472 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:36:35.472 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:36:35.473 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:36:35.473 INFO:teuthology.orchestra.run.smithi103.stdout:Install 3 Packages 2023-07-02T08:36:35.473 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:36:35.474 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 88 M 2023-07-02T08:36:35.474 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 95 M 2023-07-02T08:36:35.474 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:36:40.903 INFO:teuthology.orchestra.run.smithi164.stdout:(1/3): kernel-4.18.0-499.el8.x86_64.rpm 1.6 MB/s | 9.8 MB 00:06 2023-07-02T08:36:50.136 INFO:teuthology.orchestra.run.smithi103.stdout:(1/3): kernel-4.18.0-499.el8.x86_64.rpm 684 kB/s | 9.8 MB 00:14 2023-07-02T08:36:57.593 INFO:teuthology.orchestra.run.smithi164.stdout:(2/3): kernel-modules-4.18.0-499.el8.x86_64.rpm 1.5 MB/s | 35 MB 00:22 2023-07-02T08:36:59.994 INFO:teuthology.orchestra.run.smithi103.stdout:(2/3): kernel-core-4.18.0-499.el8.x86_64.rpm 1.7 MB/s | 43 MB 00:24 2023-07-02T08:37:21.642 INFO:teuthology.orchestra.run.smithi103.stdout:(3/3): kernel-modules-4.18.0-499.el8.x86_64.rpm 782 kB/s | 35 MB 00:46 2023-07-02T08:37:21.642 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:37:21.643 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.9 MB/s | 88 MB 00:46 2023-07-02T08:37:22.299 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:37:22.728 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:37:22.729 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:37:23.929 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:37:23.954 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:37:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:37:27.222 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-4.18.0-499.el8.x86_64 1/3 2023-07-02T08:37:30.122 INFO:teuthology.orchestra.run.smithi164.stdout:(3/3): kernel-core-4.18.0-499.el8.x86_64.rpm 789 kB/s | 43 MB 00:55 2023-07-02T08:37:30.123 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:37:30.123 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.6 MB/s | 88 MB 00:55 2023-07-02T08:37:30.788 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:37:30.837 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-4.18.0-499.el8.x86_64 1/3 2023-07-02T08:37:31.212 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:37:31.212 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:37:31.230 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-4.18.0-499.el8.x86_64 2/3 2023-07-02T08:37:32.415 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:37:32.446 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:37:34.586 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-4.18.0-499.el8.x86_64 2/3 2023-07-02T08:37:34.890 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:37:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:37:35.742 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-core-4.18.0-499.el8.x86_64 1/3 2023-07-02T08:37:39.321 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-4.18.0-499.el8.x86_64 1/3 2023-07-02T08:37:39.725 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-4.18.0-499.el8.x86_64 2/3 2023-07-02T08:37:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-4.18.0-499.el8.x86_64 2/3 2023-07-02T08:37:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:37:56.664 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:37:56.687 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:37:57.033 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:37:57.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-4.18.0-499.el8.x86_64 1/3 2023-07-02T08:37:57.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-4.18.0-499.el8.x86_64 2/3 2023-07-02T08:37:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:37:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:37:57.673 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:37:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-4.18.0-499.el8.x86_64 kernel-core-4.18.0-499.el8.x86_64 2023-07-02T08:37:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-4.18.0-499.el8.x86_64 2023-07-02T08:37:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:37:57.673 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:37:57.797 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T08:37:58.013 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-4.18.0-499.el8.x86_64 2023-07-02T08:37:58.014 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-499.el8.x86_64 2023-07-02T08:37:58.014 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-499.el8.x86_64 2023-07-02T08:37:58.014 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-02T08:37:58.015 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-02T08:37:58.015 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-07-02T08:37:58.015 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2023-07-02T08:37:58.016 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T08:37:58.224 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-4.18.0-499.el8.x86_64 2023-07-02T08:37:58.224 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-499.el8.x86_64 2023-07-02T08:37:58.224 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2023-07-02T08:37:58.258 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2023-07-02T08:37:58.259 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2023-07-02T08:37:58.259 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.02 2023-07-02T08:37:58.260 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 129.el8 2023-07-02T08:37:58.260 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2023-07-02T08:37:58.260 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-07-02T08:37:58.260 INFO:teuthology.orchestra.run.smithi103.stdout:Group : System Environment/Base 2023-07-02T08:37:58.260 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 9265586 2023-07-02T08:37:58.260 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2023-07-02T08:37:58.260 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-07-02T08:37:58.261 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-07-02T08:37:58.261 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-07-02T08:37:58.261 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : kbuilder.bsys.centos.org 2023-07-02T08:37:58.261 INFO:teuthology.orchestra.run.smithi103.stdout:Relocations : (not relocatable) 2023-07-02T08:37:58.261 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : CentOS BuildSystem 2023-07-02T08:37:58.261 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2023-07-02T08:37:58.261 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-02T08:37:58.262 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2023-07-02T08:37:58.262 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2023-07-02T08:37:58.262 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:37:58.262 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-02T08:37:58.262 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-02T08:37:58.262 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-02T08:37:58.263 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2023-07-02T08:37:58.263 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:37:58.263 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2023-07-02T08:37:58.264 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-02T08:37:58.264 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 4.18.0-499.el8.x86_64 2023-07-02T08:37:58.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-02T08:38:00.020 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2023-07-02T08:38:02.357 INFO:teuthology.orchestra.run.smithi103.stderr:done 2023-07-02T08:38:02.358 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2023-07-02T08:38:02.422 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.KoNUUVvFsR 2023-07-02T08:38:02.423 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KoNUUVvFsR 2023-07-02T08:38:02.496 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.KoNUUVvFsR 2023-07-02T08:38:02.646 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.KoNUUVvFsR is 6KB 2023-07-02T08:38:02.694 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.KoNUUVvFsR 2023-07-02T08:38:02.712 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2023-07-02T08:38:02.788 INFO:teuthology.orchestra.run.smithi103.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-07-02T08:38:02.788 INFO:teuthology.orchestra.run.smithi103.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-07-02T08:38:02.789 INFO:teuthology.orchestra.run.smithi103.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-07-02T08:38:02.789 INFO:teuthology.orchestra.run.smithi103.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-499.el8.x86_64.conf 2023-07-02T08:38:02.790 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-499.el8.x86_64 2023-07-02T08:38:02.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2023-07-02T08:38:05.098 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:38:05.121 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:38:05.479 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:38:05.480 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-4.18.0-499.el8.x86_64 1/3 2023-07-02T08:38:05.480 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-core-4.18.0-499.el8.x86_64 2/3 2023-07-02T08:38:06.169 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-4.18.0-499.el8.x86_64 3/3 2023-07-02T08:38:06.170 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:38:06.170 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:38:06.170 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-4.18.0-499.el8.x86_64 kernel-core-4.18.0-499.el8.x86_64 2023-07-02T08:38:06.170 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-4.18.0-499.el8.x86_64 2023-07-02T08:38:06.170 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:38:06.171 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:38:06.294 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T08:38:06.547 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-4.18.0-499.el8.x86_64 2023-07-02T08:38:06.548 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-499.el8.x86_64 2023-07-02T08:38:06.548 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-499.el8.x86_64 2023-07-02T08:38:06.549 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-02T08:38:06.549 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-02T08:38:06.549 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-07-02T08:38:06.549 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2023-07-02T08:38:06.550 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T08:38:06.756 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-4.18.0-499.el8.x86_64 2023-07-02T08:38:06.757 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-499.el8.x86_64 2023-07-02T08:38:06.758 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2023-07-02T08:38:06.792 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2023-07-02T08:38:06.792 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2023-07-02T08:38:06.793 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.02 2023-07-02T08:38:06.793 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 129.el8 2023-07-02T08:38:06.793 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2023-07-02T08:38:06.793 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-07-02T08:38:06.794 INFO:teuthology.orchestra.run.smithi164.stdout:Group : System Environment/Base 2023-07-02T08:38:06.794 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 9265586 2023-07-02T08:38:06.794 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2023-07-02T08:38:06.794 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-07-02T08:38:06.795 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-07-02T08:38:06.795 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-07-02T08:38:06.796 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : kbuilder.bsys.centos.org 2023-07-02T08:38:06.796 INFO:teuthology.orchestra.run.smithi164.stdout:Relocations : (not relocatable) 2023-07-02T08:38:06.796 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : CentOS BuildSystem 2023-07-02T08:38:06.796 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2023-07-02T08:38:06.797 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-02T08:38:06.797 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2023-07-02T08:38:06.797 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2023-07-02T08:38:06.797 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:38:06.797 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-02T08:38:06.797 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-02T08:38:06.798 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-02T08:38:06.798 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2023-07-02T08:38:06.798 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:38:06.798 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2023-07-02T08:38:06.800 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-02T08:38:06.800 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 4.18.0-499.el8.x86_64 2023-07-02T08:38:06.801 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-02T08:38:08.423 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2023-07-02T08:38:10.324 INFO:teuthology.orchestra.run.smithi164.stderr:done 2023-07-02T08:38:10.326 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2023-07-02T08:38:10.392 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.2Ge1Fm41GY 2023-07-02T08:38:10.392 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2Ge1Fm41GY 2023-07-02T08:38:10.427 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.2Ge1Fm41GY 2023-07-02T08:38:10.573 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.2Ge1Fm41GY is 6KB 2023-07-02T08:38:10.623 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.2Ge1Fm41GY 2023-07-02T08:38:10.641 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2023-07-02T08:38:10.713 INFO:teuthology.orchestra.run.smithi164.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-07-02T08:38:10.714 INFO:teuthology.orchestra.run.smithi164.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-07-02T08:38:10.714 INFO:teuthology.orchestra.run.smithi164.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-07-02T08:38:10.714 INFO:teuthology.orchestra.run.smithi164.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-499.el8.x86_64.conf 2023-07-02T08:38:10.714 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-499.el8.x86_64 2023-07-02T08:38:10.865 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2023-07-02T08:38:32.972 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-07-02T08:38:32.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-07-02T08:38:32.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:38:40.876 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-07-02T08:38:40.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-07-02T08:38:40.877 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:39:07.633 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2023-07-02T08:39:15.569 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-07-02T08:39:16.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-07-02T08:39:16.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:39:19.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2023-07-02T08:39:24.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-07-02T08:39:24.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:39:31.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-07-02T08:39:31.705 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:39:42.963 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-07-02T08:39:50.129 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2023-07-02T08:39:54.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-07-02T08:39:54.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:39:56.325 DEBUG:teuthology.orchestra.run.smithi164:> true 2023-07-02T08:39:56.827 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-07-02T08:39:56.827 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "4.18.0-499.el8.x86_64"... 2023-07-02T08:39:56.828 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2023-07-02T08:39:56.849 INFO:teuthology.orchestra.run.smithi164.stdout:4.18.0-499.el8.x86_64 2023-07-02T08:39:56.849 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-499.el8.x86_64 vs 4.18.0-499.el8.x86_64 2023-07-02T08:39:56.850 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-02T08:39:56.850 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-07-02T08:39:57.851 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-07-02T08:39:57.852 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-02T08:39:57.936 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2023-07-02T08:39:57.951 DEBUG:teuthology.parallel:result is None 2023-07-02T08:40:05.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-07-02T08:40:05.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:40:05.846 DEBUG:teuthology.orchestra.run.smithi103:> true 2023-07-02T08:40:06.913 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-07-02T08:40:06.914 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-499.el8.x86_64"... 2023-07-02T08:40:06.914 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2023-07-02T08:40:06.936 INFO:teuthology.orchestra.run.smithi103.stdout:4.18.0-499.el8.x86_64 2023-07-02T08:40:06.936 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-499.el8.x86_64 vs 4.18.0-499.el8.x86_64 2023-07-02T08:40:06.936 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-02T08:40:06.936 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-07-02T08:40:07.937 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-02T08:40:07.938 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-02T08:40:08.001 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2023-07-02T08:40:08.016 DEBUG:teuthology.parallel:result is None 2023-07-02T08:40:08.017 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-02T08:40:08.039 INFO:teuthology.task.internal:Creating test directory... 2023-07-02T08:40:08.040 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-02T08:40:08.057 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-02T08:40:08.087 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-02T08:40:08.096 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-02T08:40:08.133 INFO:teuthology.task.internal:Creating archive directory... 2023-07-02T08:40:08.134 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-02T08:40:08.144 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-02T08:40:08.183 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-02T08:40:08.193 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-02T08:40:08.194 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-02T08:40:08.232 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-02T08:40:08.259 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T08:40:08.269 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T08:40:08.279 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T08:40:08.293 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T08:40:08.294 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-02T08:40:08.313 INFO:teuthology.task.internal:Configuring sudo... 2023-07-02T08:40:08.314 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-02T08:40:08.320 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-02T08:40:08.353 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-02T08:40:08.378 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-02T08:40:08.379 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-02T08:40:08.396 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-02T08:40:08.416 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T08:40:08.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T08:40:08.557 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T08:40:08.654 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T08:40:08.690 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:40:08.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-02T08:40:08.759 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T08:40:08.817 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T08:40:08.870 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T08:40:08.965 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T08:40:09.000 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:40:09.001 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-02T08:40:09.069 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2023-07-02T08:40:09.072 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2023-07-02T08:40:09.137 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T08:40:09.170 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T08:40:09.646 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-02T08:40:09.659 INFO:teuthology.task.internal:Starting timer... 2023-07-02T08:40:09.659 INFO:teuthology.run_tasks:Running task pcp... 2023-07-02T08:40:09.695 INFO:teuthology.run_tasks:Running task selinux... 2023-07-02T08:40:09.736 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-02T08:40:09.738 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2023-07-02T08:40:09.822 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2023-07-02T08:40:09.825 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2023-07-02T08:40:09.926 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2023-07-02T08:40:09.927 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-02T08:40:09.927 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2023-07-02T08:40:09.954 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2023-07-02T08:40:09.955 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2023-07-02T08:40:09.995 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2023-07-02T08:40:09.995 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2023-07-02T08:40:09.996 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T08:40:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T08:40:10.024 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T08:40:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T08:40:10.065 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-02T08:40:10.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2023-07-02T08:40:10.105 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2023-07-02T08:40:10.134 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-02T08:40:10.148 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-07-02T08:40:10.158 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-02T08:40:10.205 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-02T08:40:10.207 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi103.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-02T08:47:47.352 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2023-07-02T08:47:47.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-07-02T08:47:47.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:47:47.425 DEBUG:teuthology.orchestra.run.smithi103:> true 2023-07-02T08:47:47.494 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-07-02T08:47:47.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-07-02T08:47:47.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T08:47:47.565 DEBUG:teuthology.orchestra.run.smithi164:> true 2023-07-02T08:47:47.638 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-07-02T08:47:47.639 INFO:teuthology.run_tasks:Running task clock... 2023-07-02T08:47:47.651 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-02T08:47:47.652 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-02T08:47:47.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T08:47:47.654 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-02T08:47:47.655 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T08:47:47.685 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-02T08:47:47.700 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-02T08:47:47.715 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-02T08:47:47.730 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-02T08:47:47.760 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2023-07-02T08:47:47.771 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2023-07-02T08:47:47.786 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-02T08:47:47.794 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2023-07-02T08:47:47.799 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-02T08:47:47.806 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2023-07-02T08:47:47.820 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-02T08:47:47.834 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-02T08:47:47.878 INFO:teuthology.orchestra.run.smithi103.stderr:bash: ntpq: command not found 2023-07-02T08:47:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T08:47:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2023-07-02T08:47:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T08:47:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T08:47:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T08:47:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T08:47:47.911 INFO:teuthology.orchestra.run.smithi164.stderr:bash: ntpq: command not found 2023-07-02T08:47:47.914 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T08:47:47.915 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2023-07-02T08:47:47.915 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T08:47:47.915 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T08:47:47.915 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T08:47:47.915 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T08:47:47.916 INFO:teuthology.run_tasks:Running task install... 2023-07-02T08:47:47.924 DEBUG:teuthology.task.install:project ceph 2023-07-02T08:47:47.925 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '6450d2709f998063da1a187d826fea699691b0b1'}} 2023-07-02T08:47:47.925 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '6450d2709f998063da1a187d826fea699691b0b1'} 2023-07-02T08:47:47.925 INFO:teuthology.task.install:Using flavor: crimson 2023-07-02T08:47:47.931 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-02T08:47:47.931 INFO:teuthology.task.install:extra packages: [] 2023-07-02T08:47:47.932 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6450d2709f998063da1a187d826fea699691b0b1', 'tag': None, 'wait_for_package': False} 2023-07-02T08:47:47.933 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=6450d2709f998063da1a187d826fea699691b0b1 2023-07-02T08:47:47.935 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6450d2709f998063da1a187d826fea699691b0b1', 'tag': None, 'wait_for_package': False} 2023-07-02T08:47:47.935 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=6450d2709f998063da1a187d826fea699691b0b1 2023-07-02T08:47:48.077 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yingxin-msgr-multi-core-crimson-10/6450d2709f998063da1a187d826fea699691b0b1/centos/8/flavors/crimson/ 2023-07-02T08:47:48.078 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4632.g6450d270 2023-07-02T08:47:48.145 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yingxin-msgr-multi-core-crimson-10/6450d2709f998063da1a187d826fea699691b0b1/centos/8/flavors/crimson/ 2023-07-02T08:47:48.146 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4632.g6450d270 2023-07-02T08:47:48.376 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-msgr-multi-core-crimson-10/6450d2709f998063da1a187d826fea699691b0b1/centos/8/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-msgr-multi-core-crimson-10/6450d2709f998063da1a187d826fea699691b0b1/centos/8/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-msgr-multi-core-crimson-10/6450d2709f998063da1a187d826fea699691b0b1/centos/8/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-02T08:47:48.376 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:47:48.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-02T08:47:48.406 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-07-02T08:47:48.406 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6450d2709f998063da1a187d826fea699691b0b1/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-02T08:47:48.433 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-msgr-multi-core-crimson-10/6450d2709f998063da1a187d826fea699691b0b1/centos/8/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-msgr-multi-core-crimson-10/6450d2709f998063da1a187d826fea699691b0b1/centos/8/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-msgr-multi-core-crimson-10/6450d2709f998063da1a187d826fea699691b0b1/centos/8/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-02T08:47:48.433 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:47:48.434 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-02T08:47:48.464 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-07-02T08:47:48.465 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6450d2709f998063da1a187d826fea699691b0b1/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-02T08:47:48.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-02T08:47:48.534 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-02T08:47:48.553 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-02T08:47:48.614 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-02T08:47:48.619 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2023-07-02T08:47:48.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2023-07-02T08:47:48.682 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2023-07-02T08:47:48.683 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2023-07-02T08:47:48.889 INFO:teuthology.orchestra.run.smithi103.stdout:73 files removed 2023-07-02T08:47:48.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2023-07-02T08:47:48.954 INFO:teuthology.orchestra.run.smithi164.stdout:73 files removed 2023-07-02T08:47:48.982 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw 2023-07-02T08:47:50.156 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 110 kB/s | 84 kB 00:00 2023-07-02T08:47:50.326 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 84 kB/s | 84 kB 00:00 2023-07-02T08:47:50.998 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 12 kB/s | 9.4 kB 00:00 2023-07-02T08:47:51.160 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 12 kB/s | 9.4 kB 00:00 2023-07-02T08:47:51.723 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 2.3 kB/s | 1.6 kB 00:00 2023-07-02T08:47:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 2.1 kB/s | 1.6 kB 00:00 2023-07-02T08:47:54.448 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - AppStream 11 MB/s | 30 MB 00:02 2023-07-02T08:47:54.511 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - AppStream 12 MB/s | 30 MB 00:02 2023-07-02T08:48:02.170 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - BaseOS 20 MB/s | 40 MB 00:01 2023-07-02T08:48:02.690 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 40 MB 00:02 2023-07-02T08:48:09.074 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2023-07-02T08:48:09.858 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 6.6 kB 00:00 2023-07-02T08:48:11.287 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - Extras 5.3 kB/s | 18 kB 00:03 2023-07-02T08:48:11.693 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 8 - PowerTools 3.6 MB/s | 6.1 MB 00:01 2023-07-02T08:48:11.971 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.6 kB 00:00 2023-07-02T08:48:12.689 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.1 MB 00:00 2023-07-02T08:48:15.470 INFO:teuthology.orchestra.run.smithi164.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2023-07-02T08:48:16.532 INFO:teuthology.orchestra.run.smithi103.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2023-07-02T08:48:17.605 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 16 MB 00:01 2023-07-02T08:48:17.733 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-07-02T08:48:21.361 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 392 kB/s | 24 kB 00:00 2023-07-02T08:48:21.500 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 235 kB/s | 24 kB 00:00 2023-07-02T08:48:25.360 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:48:25.363 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:48:25.363 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:48:25.363 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:48:25.363 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:48:25.363 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:18.0.0-4632.g6450d270.el8 ceph 15 M 2023-07-02T08:48:25.363 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2023-07-02T08:48:25.364 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 3.3 M 2023-07-02T08:48:25.364 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 3.6 M 2023-07-02T08:48:25.364 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:48:25.364 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:18.0.0-4632.g6450d270.el8 ceph 6.8 M 2023-07-02T08:48:25.364 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:18.0.0-4632.g6450d270.el8 ceph 23 M 2023-07-02T08:48:25.364 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:18.0.0-4632.g6450d270.el8 ceph 24 k 2023-07-02T08:48:25.364 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-02T08:48:25.365 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 747 k 2023-07-02T08:48:25.365 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-02T08:48:25.365 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-07-02T08:48:25.365 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 462 k 2023-07-02T08:48:25.365 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-07-02T08:48:25.365 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 7.4 M 2023-07-02T08:48:25.365 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-02T08:48:25.365 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-07-02T08:48:25.366 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:18.0.0-4632.g6450d270.el8 ceph 45 k 2023-07-02T08:48:25.366 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:18.0.0-4632.g6450d270.el8 ceph 119 k 2023-07-02T08:48:25.366 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:18.0.0-4632.g6450d270.el8 ceph 197 k 2023-07-02T08:48:25.366 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-07-02T08:48:25.366 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:18.0.0-4632.g6450d270.el8 ceph 372 k 2023-07-02T08:48:25.366 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:18.0.0-4632.g6450d270.el8 ceph 364 k 2023-07-02T08:48:25.366 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:18.0.0-4632.g6450d270.el8 ceph 107 k 2023-07-02T08:48:25.367 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-02T08:48:25.367 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:48:25.367 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:48:25.367 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:48:25.367 INFO:teuthology.orchestra.run.smithi164.stdout:Install 21 Packages 2023-07-02T08:48:25.367 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 2 Packages 2023-07-02T08:48:25.367 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:48:25.368 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 64 M 2023-07-02T08:48:25.368 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:48:25.437 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:48:25.440 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:48:25.440 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:48:25.440 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:48:25.440 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:48:25.440 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:18.0.0-4632.g6450d270.el8 ceph 15 M 2023-07-02T08:48:25.441 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2023-07-02T08:48:25.441 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 3.3 M 2023-07-02T08:48:25.441 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 3.6 M 2023-07-02T08:48:25.441 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:48:25.441 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:18.0.0-4632.g6450d270.el8 ceph 6.8 M 2023-07-02T08:48:25.441 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:18.0.0-4632.g6450d270.el8 ceph 23 M 2023-07-02T08:48:25.441 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:18.0.0-4632.g6450d270.el8 ceph 24 k 2023-07-02T08:48:25.442 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-02T08:48:25.442 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 747 k 2023-07-02T08:48:25.442 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-02T08:48:25.442 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-07-02T08:48:25.442 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 462 k 2023-07-02T08:48:25.442 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-07-02T08:48:25.442 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:18.0.0-4632.g6450d270.el8 ceph 7.4 M 2023-07-02T08:48:25.442 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-02T08:48:25.443 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-07-02T08:48:25.443 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:18.0.0-4632.g6450d270.el8 ceph 45 k 2023-07-02T08:48:25.443 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:18.0.0-4632.g6450d270.el8 ceph 119 k 2023-07-02T08:48:25.443 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:18.0.0-4632.g6450d270.el8 ceph 197 k 2023-07-02T08:48:25.443 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-07-02T08:48:25.443 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:18.0.0-4632.g6450d270.el8 ceph 372 k 2023-07-02T08:48:25.443 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:18.0.0-4632.g6450d270.el8 ceph 364 k 2023-07-02T08:48:25.444 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:18.0.0-4632.g6450d270.el8 ceph 107 k 2023-07-02T08:48:25.444 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-02T08:48:25.444 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:48:25.444 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:48:25.444 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:48:25.444 INFO:teuthology.orchestra.run.smithi103.stdout:Install 21 Packages 2023-07-02T08:48:25.444 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2023-07-02T08:48:25.444 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:48:25.445 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 64 M 2023-07-02T08:48:25.445 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:48:39.086 INFO:teuthology.orchestra.run.smithi164.stdout:(1/23): ceph-base-18.0.0-4632.g6450d270.el8.x86 528 kB/s | 6.8 MB 00:13 2023-07-02T08:48:39.210 INFO:teuthology.orchestra.run.smithi164.stdout:(2/23): ceph-selinux-18.0.0-4632.g6450d270.el8. 198 kB/s | 24 kB 00:00 2023-07-02T08:48:39.367 INFO:teuthology.orchestra.run.smithi103.stdout:(1/23): ceph-base-18.0.0-4632.g6450d270.el8.x86 514 kB/s | 6.8 MB 00:13 2023-07-02T08:48:39.476 INFO:teuthology.orchestra.run.smithi103.stdout:(2/23): ceph-selinux-18.0.0-4632.g6450d270.el8. 226 kB/s | 24 kB 00:00 2023-07-02T08:48:40.453 INFO:teuthology.orchestra.run.smithi164.stdout:(3/23): libcephfs2-18.0.0-4632.g6450d270.el8.x8 601 kB/s | 747 kB 00:01 2023-07-02T08:48:41.170 INFO:teuthology.orchestra.run.smithi164.stdout:(4/23): libradosstriper1-18.0.0-4632.g6450d270. 644 kB/s | 462 kB 00:00 2023-07-02T08:48:41.826 INFO:teuthology.orchestra.run.smithi103.stdout:(3/23): libcephfs2-18.0.0-4632.g6450d270.el8.x8 318 kB/s | 747 kB 00:02 2023-07-02T08:48:43.117 INFO:teuthology.orchestra.run.smithi103.stdout:(4/23): libradosstriper1-18.0.0-4632.g6450d270. 358 kB/s | 462 kB 00:01 2023-07-02T08:48:49.145 INFO:teuthology.orchestra.run.smithi164.stdout:(5/23): librgw2-18.0.0-4632.g6450d270.el8.x86_6 947 kB/s | 7.4 MB 00:07 2023-07-02T08:48:49.253 INFO:teuthology.orchestra.run.smithi164.stdout:(6/23): python3-ceph-argparse-18.0.0-4632.g6450 416 kB/s | 45 kB 00:00 2023-07-02T08:48:49.445 INFO:teuthology.orchestra.run.smithi164.stdout:(7/23): python3-ceph-common-18.0.0-4632.g6450d2 620 kB/s | 119 kB 00:00 2023-07-02T08:48:49.812 INFO:teuthology.orchestra.run.smithi164.stdout:(8/23): python3-cephfs-18.0.0-4632.g6450d270.el 538 kB/s | 197 kB 00:00 2023-07-02T08:48:50.479 INFO:teuthology.orchestra.run.smithi164.stdout:(9/23): python3-rados-18.0.0-4632.g6450d270.el8 558 kB/s | 372 kB 00:00 2023-07-02T08:48:51.146 INFO:teuthology.orchestra.run.smithi164.stdout:(10/23): python3-rbd-18.0.0-4632.g6450d270.el8. 547 kB/s | 364 kB 00:00 2023-07-02T08:48:51.554 INFO:teuthology.orchestra.run.smithi164.stdout:(11/23): python3-rgw-18.0.0-4632.g6450d270.el8. 261 kB/s | 107 kB 00:00 2023-07-02T08:48:52.021 INFO:teuthology.orchestra.run.smithi164.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 758 kB/s | 354 kB 00:00 2023-07-02T08:48:52.122 INFO:teuthology.orchestra.run.smithi164.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2023-07-02T08:48:52.214 INFO:teuthology.orchestra.run.smithi164.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 480 kB/s | 44 kB 00:00 2023-07-02T08:48:52.597 INFO:teuthology.orchestra.run.smithi164.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 122 kB/s | 47 kB 00:00 2023-07-02T08:48:52.764 INFO:teuthology.orchestra.run.smithi164.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2023-07-02T08:48:52.806 INFO:teuthology.orchestra.run.smithi164.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-07-02T08:48:52.857 INFO:teuthology.orchestra.run.smithi164.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2023-07-02T08:48:53.223 INFO:teuthology.orchestra.run.smithi164.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 4.7 MB/s | 1.7 MB 00:00 2023-07-02T08:48:54.382 INFO:teuthology.orchestra.run.smithi164.stdout:(20/23): ceph-radosgw-18.0.0-4632.g6450d270.el8 531 kB/s | 15 MB 00:28 2023-07-02T08:48:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:(5/23): ceph-common-18.0.0-4632.g6450d270.el8.x 779 kB/s | 23 MB 00:30 2023-07-02T08:48:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:(6/23): python3-ceph-argparse-18.0.0-4632.g6450 448 kB/s | 45 kB 00:00 2023-07-02T08:48:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:(7/23): python3-ceph-common-18.0.0-4632.g6450d2 339 kB/s | 119 kB 00:00 2023-07-02T08:48:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:(8/23): python3-cephfs-18.0.0-4632.g6450d270.el 623 kB/s | 197 kB 00:00 2023-07-02T08:48:57.893 INFO:teuthology.orchestra.run.smithi103.stdout:(9/23): librgw2-18.0.0-4632.g6450d270.el8.x86_6 511 kB/s | 7.4 MB 00:14 2023-07-02T08:48:57.919 INFO:teuthology.orchestra.run.smithi103.stdout:(10/23): python3-rados-18.0.0-4632.g6450d270.el 686 kB/s | 372 kB 00:00 2023-07-02T08:48:58.094 INFO:teuthology.orchestra.run.smithi103.stdout:(11/23): python3-rgw-18.0.0-4632.g6450d270.el8. 610 kB/s | 107 kB 00:00 2023-07-02T08:48:58.161 INFO:teuthology.orchestra.run.smithi103.stdout:(12/23): ceph-radosgw-18.0.0-4632.g6450d270.el8 467 kB/s | 15 MB 00:32 2023-07-02T08:48:58.553 INFO:teuthology.orchestra.run.smithi103.stdout:(13/23): python3-rbd-18.0.0-4632.g6450d270.el8. 553 kB/s | 364 kB 00:00 2023-07-02T08:48:58.620 INFO:teuthology.orchestra.run.smithi103.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 673 kB/s | 354 kB 00:00 2023-07-02T08:48:58.645 INFO:teuthology.orchestra.run.smithi103.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 536 kB/s | 259 kB 00:00 2023-07-02T08:48:58.706 INFO:teuthology.orchestra.run.smithi164.stdout:(21/23): librbd1-18.0.0-4632.g6450d270.el8.x86_ 841 kB/s | 3.6 MB 00:04 2023-07-02T08:48:58.807 INFO:teuthology.orchestra.run.smithi164.stdout:(22/23): librados2-18.0.0-4632.g6450d270.el8.x8 611 kB/s | 3.3 MB 00:05 2023-07-02T08:48:58.879 INFO:teuthology.orchestra.run.smithi103.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-07-02T08:48:58.896 INFO:teuthology.orchestra.run.smithi103.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 128 kB/s | 44 kB 00:00 2023-07-02T08:48:58.938 INFO:teuthology.orchestra.run.smithi103.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-07-02T08:48:59.022 INFO:teuthology.orchestra.run.smithi103.stdout:(19/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 117 kB/s | 47 kB 00:00 2023-07-02T08:48:59.080 INFO:teuthology.orchestra.run.smithi103.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 408 kB/s | 75 kB 00:00 2023-07-02T08:48:59.122 INFO:teuthology.orchestra.run.smithi103.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 9.4 MB/s | 1.7 MB 00:00 2023-07-02T08:49:02.664 INFO:teuthology.orchestra.run.smithi103.stdout:(22/23): librbd1-18.0.0-4632.g6450d270.el8.x86_ 1.0 MB/s | 3.6 MB 00:03 2023-07-02T08:49:03.022 INFO:teuthology.orchestra.run.smithi103.stdout:(23/23): librados2-18.0.0-4632.g6450d270.el8.x8 852 kB/s | 3.3 MB 00:03 2023-07-02T08:49:03.024 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:49:03.025 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.7 MB/s | 64 MB 00:37 2023-07-02T08:49:03.064 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:49:03.110 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:49:03.110 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:49:03.490 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:49:03.493 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:49:04.123 INFO:teuthology.orchestra.run.smithi164.stdout:(23/23): ceph-common-18.0.0-4632.g6450d270.el8. 628 kB/s | 23 MB 00:38 2023-07-02T08:49:04.125 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:49:04.126 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.7 MB/s | 64 MB 00:38 2023-07-02T08:49:04.163 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:49:04.209 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:49:04.209 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:49:04.578 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:49:04.581 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:49:04.598 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:49:04.807 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-07-02T08:49:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-07-02T08:49:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-07-02T08:49:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:18.0.0-4632.g6450d270.el8.x86_64 3/25 2023-07-02T08:49:05.614 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:18.0.0-4632.g6450d270.el8.x86_64 3/25 2023-07-02T08:49:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:49:05.923 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:18.0.0-4632.g6450d270.el8.x86_64 4/25 2023-07-02T08:49:05.979 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-07-02T08:49:06.117 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 5/25 2023-07-02T08:49:06.156 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-07-02T08:49:06.514 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 5/25 2023-07-02T08:49:06.547 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-07-02T08:49:06.692 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 6/25 2023-07-02T08:49:06.706 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:18.0.0-4632.g6450d270.el8.x86_64 3/25 2023-07-02T08:49:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:18.0.0-4632.g6450d270.el8.x86_64 3/25 2023-07-02T08:49:06.898 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 6/25 2023-07-02T08:49:07.129 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:18.0.0-4632.g6450d270.el8.x86_64 4/25 2023-07-02T08:49:07.238 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-07-02T08:49:07.315 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 5/25 2023-07-02T08:49:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-07-02T08:49:07.658 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-07-02T08:49:07.725 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-07-02T08:49:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 5/25 2023-07-02T08:49:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:18.0.0-4632.g6450d270.el8. 10/25 2023-07-02T08:49:07.889 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 6/25 2023-07-02T08:49:08.093 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:18.0.0-4632.g6450d270.el8.x86_64 11/25 2023-07-02T08:49:08.123 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 6/25 2023-07-02T08:49:08.335 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-07-02T08:49:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:18.0.0-4632.g6450d270.el8.x86_64 12/25 2023-07-02T08:49:08.475 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:18.0.0-4632.g6450d270.el8.x86_6 13/25 2023-07-02T08:49:08.555 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4632.g6450d270.el8.x86_6 13/25 2023-07-02T08:49:08.564 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-07-02T08:49:08.767 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-07-02T08:49:08.781 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-07-02T08:49:08.833 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-07-02T08:49:09.034 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:18.0.0-4632.g6450d270.el8. 10/25 2023-07-02T08:49:09.266 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:18.0.0-4632.g6450d270.el8.x86_64 11/25 2023-07-02T08:49:09.491 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:18.0.0-4632.g6450d270.el8.x86_64 12/25 2023-07-02T08:49:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:18.0.0-4632.g6450d270.el8.x86_6 13/25 2023-07-02T08:49:09.703 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-07-02T08:49:09.721 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4632.g6450d270.el8.x86_6 13/25 2023-07-02T08:49:09.916 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:18.0.0-4632.g6450d270.el8.x86_64 16/25 2023-07-02T08:49:09.985 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:18.0.0-4632.g6450d270.el8.x86_64 16/25 2023-07-02T08:49:09.986 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-07-02T08:49:10.166 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:18.0.0-4632.g6450d270.el8.x86_64 17/25 2023-07-02T08:49:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-07-02T08:49:10.533 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:18.0.0-4632.g6450d270.el8.x8 19/25 2023-07-02T08:49:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-07-02T08:49:11.205 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:18.0.0-4632.g6450d270.el8.x86_64 16/25 2023-07-02T08:49:11.267 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:18.0.0-4632.g6450d270.el8.x86_64 16/25 2023-07-02T08:49:11.463 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:18.0.0-4632.g6450d270.el8.x86_64 17/25 2023-07-02T08:49:11.639 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-07-02T08:49:11.856 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:18.0.0-4632.g6450d270.el8.x8 19/25 2023-07-02T08:49:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 20/25 2023-07-02T08:49:13.650 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 20/25 2023-07-02T08:49:14.440 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 20/25 2023-07-02T08:49:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 21/25 2023-07-02T08:49:14.826 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 21/25 2023-07-02T08:49:14.856 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 20/25 2023-07-02T08:49:14.991 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:18.0.0-4632.g6450d270.el8.x86_64 22/25 2023-07-02T08:49:15.039 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 20/25 2023-07-02T08:49:15.842 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 20/25 2023-07-02T08:49:16.117 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 21/25 2023-07-02T08:49:16.229 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 21/25 2023-07-02T08:49:16.413 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:18.0.0-4632.g6450d270.el8.x86_64 22/25 2023-07-02T08:49:34.342 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4632.g6450d270.el8.x86_64 22/25 2023-07-02T08:49:34.342 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2023-07-02T08:49:34.343 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2023-07-02T08:49:34.343 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /dev 2023-07-02T08:49:34.343 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /run 2023-07-02T08:49:34.343 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2023-07-02T08:49:34.343 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2023-07-02T08:49:34.343 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2023-07-02T08:49:34.344 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2023-07-02T08:49:34.344 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /dev 2023-07-02T08:49:34.344 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:49:35.844 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4632.g6450d270.el8.x86_64 22/25 2023-07-02T08:49:35.844 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2023-07-02T08:49:35.845 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2023-07-02T08:49:35.845 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /dev 2023-07-02T08:49:35.845 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /run 2023-07-02T08:49:35.845 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2023-07-02T08:49:35.845 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2023-07-02T08:49:35.845 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2023-07-02T08:49:35.845 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2023-07-02T08:49:35.846 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /dev 2023-07-02T08:49:35.846 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:49:36.016 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:18.0.0-4632.g6450d270.el8.x86_64 23/25 2023-07-02T08:49:36.036 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4632.g6450d270.el8.x86_64 23/25 2023-07-02T08:49:36.037 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-07-02T08:49:36.249 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-07-02T08:49:36.249 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-07-02T08:49:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-07-02T08:49:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 1/25 2023-07-02T08:49:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 2/25 2023-07-02T08:49:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:18.0.0-4632.g6450d270.el8.x86_64 3/25 2023-07-02T08:49:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:18.0.0-4632.g6450d270.el8.x86_64 4/25 2023-07-02T08:49:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 5/25 2023-07-02T08:49:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:18.0.0-4632.g6450d270.el8.x86_6 6/25 2023-07-02T08:49:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:18.0.0-4632.g6450d270.el8.x86_64 7/25 2023-07-02T08:49:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4632.g6450d270.el8. 8/25 2023-07-02T08:49:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:18.0.0-4632.g6450d270.el8.x8 9/25 2023-07-02T08:49:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:18.0.0-4632.g6450d270.el8.x86_64 10/25 2023-07-02T08:49:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:18.0.0-4632.g6450d270.el8.x86_64 11/25 2023-07-02T08:49:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:18.0.0-4632.g6450d270.el8.x86_64 12/25 2023-07-02T08:49:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:18.0.0-4632.g6450d270.el8.x86_64 13/25 2023-07-02T08:49:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-07-02T08:49:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-07-02T08:49:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-07-02T08:49:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-07-02T08:49:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-07-02T08:49:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-07-02T08:49:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-07-02T08:49:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-07-02T08:49:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:18.0.0-4632.g6450d270.el8.x86_64 22/25 2023-07-02T08:49:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-07-02T08:49:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 24/25 2023-07-02T08:49:37.480 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:18.0.0-4632.g6450d270.el8.x86_64 23/25 2023-07-02T08:49:37.500 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4632.g6450d270.el8.x86_64 23/25 2023-07-02T08:49:37.501 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-07-02T08:49:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-07-02T08:49:37.737 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-07-02T08:49:38.052 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-07-02T08:49:38.052 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:49:38.052 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2023-07-02T08:49:38.053 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.053 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.053 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:49:38.053 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.053 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.053 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.054 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.054 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-02T08:49:38.054 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.054 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-02T08:49:38.054 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-07-02T08:49:38.054 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.054 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-02T08:49:38.054 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.055 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-02T08:49:38.055 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-02T08:49:38.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-02T08:49:38.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:38.056 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-02T08:49:38.056 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:49:38.056 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:49:38.338 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2023-07-02T08:49:38.484 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-07-02T08:49:38.484 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 1/25 2023-07-02T08:49:38.484 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 2/25 2023-07-02T08:49:38.485 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:18.0.0-4632.g6450d270.el8.x86_64 3/25 2023-07-02T08:49:38.485 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:18.0.0-4632.g6450d270.el8.x86_64 4/25 2023-07-02T08:49:38.485 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 5/25 2023-07-02T08:49:38.485 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:18.0.0-4632.g6450d270.el8.x86_6 6/25 2023-07-02T08:49:38.485 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:18.0.0-4632.g6450d270.el8.x86_64 7/25 2023-07-02T08:49:38.485 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4632.g6450d270.el8. 8/25 2023-07-02T08:49:38.485 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:18.0.0-4632.g6450d270.el8.x8 9/25 2023-07-02T08:49:38.486 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:18.0.0-4632.g6450d270.el8.x86_64 10/25 2023-07-02T08:49:38.486 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:18.0.0-4632.g6450d270.el8.x86_64 11/25 2023-07-02T08:49:38.486 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:18.0.0-4632.g6450d270.el8.x86_64 12/25 2023-07-02T08:49:38.486 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:18.0.0-4632.g6450d270.el8.x86_64 13/25 2023-07-02T08:49:38.486 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-07-02T08:49:38.486 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-07-02T08:49:38.486 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-07-02T08:49:38.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-07-02T08:49:38.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-07-02T08:49:38.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-07-02T08:49:38.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-07-02T08:49:38.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-07-02T08:49:38.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:18.0.0-4632.g6450d270.el8.x86_64 22/25 2023-07-02T08:49:38.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-07-02T08:49:38.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 24/25 2023-07-02T08:49:39.169 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:18 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:49:39.678 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-07-02T08:49:39.679 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:49:39.679 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2023-07-02T08:49:39.679 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.679 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.679 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:49:39.680 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.680 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.680 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.680 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.680 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-02T08:49:39.680 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.680 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-02T08:49:39.681 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-07-02T08:49:39.681 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.681 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-02T08:49:39.681 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.681 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-02T08:49:39.681 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-02T08:49:39.681 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.682 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.682 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.682 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-02T08:49:39.682 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.682 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.682 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:49:39.682 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-02T08:49:39.682 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:49:39.683 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:49:39.960 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-test 2023-07-02T08:49:40.483 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:49:40.484 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:49:40.484 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:49:40.484 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:49:40.485 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:49:40.485 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:18.0.0-4632.g6450d270.el8 ceph 150 M 2023-07-02T08:49:40.485 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:49:40.485 INFO:teuthology.orchestra.run.smithi103.stdout: cryptopp x86_64 8.6.0-1.el8 epel 1.3 M 2023-07-02T08:49:40.485 INFO:teuthology.orchestra.run.smithi103.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-07-02T08:49:40.485 INFO:teuthology.orchestra.run.smithi103.stdout: libasan6 x86_64 11.2.1-9.1.el8 appstream 402 k 2023-07-02T08:49:40.485 INFO:teuthology.orchestra.run.smithi103.stdout: libatomic x86_64 8.5.0-20.el8 baseos 26 k 2023-07-02T08:49:40.486 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:18.0.0-4632.g6450d270.el8 ceph 135 k 2023-07-02T08:49:40.486 INFO:teuthology.orchestra.run.smithi103.stdout: libubsan x86_64 8.5.0-20.el8 baseos 152 k 2023-07-02T08:49:40.486 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-07-02T08:49:40.486 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-07-02T08:49:40.486 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-07-02T08:49:40.486 INFO:teuthology.orchestra.run.smithi103.stdout: yaml-cpp x86_64 0.6.2-3.el8 epel 171 k 2023-07-02T08:49:40.487 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:49:40.487 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:49:40.487 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:49:40.487 INFO:teuthology.orchestra.run.smithi103.stdout:Install 11 Packages 2023-07-02T08:49:40.487 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:49:40.488 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 153 M 2023-07-02T08:49:40.488 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.7 G 2023-07-02T08:49:40.488 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:49:40.794 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:19 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:49:41.133 INFO:teuthology.orchestra.run.smithi103.stdout:(1/11): jq-1.6-7.el8.x86_64.rpm 517 kB/s | 202 kB 00:00 2023-07-02T08:49:41.317 INFO:teuthology.orchestra.run.smithi103.stdout:(2/11): libasan6-11.2.1-9.1.el8.x86_64.rpm 2.1 MB/s | 402 kB 00:00 2023-07-02T08:49:41.417 INFO:teuthology.orchestra.run.smithi103.stdout:(3/11): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-07-02T08:49:41.517 INFO:teuthology.orchestra.run.smithi103.stdout:(4/11): libcephsqlite-18.0.0-4632.g6450d270.el8 174 kB/s | 135 kB 00:00 2023-07-02T08:49:41.543 INFO:teuthology.orchestra.run.smithi103.stdout:(5/11): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2023-07-02T08:49:41.843 INFO:teuthology.orchestra.run.smithi103.stdout:(6/11): xmlstarlet-1.6.1-20.el8.x86_64.rpm 217 kB/s | 71 kB 00:00 2023-07-02T08:49:41.868 INFO:teuthology.orchestra.run.smithi103.stdout:(7/11): libatomic-8.5.0-20.el8.x86_64.rpm 79 kB/s | 26 kB 00:00 2023-07-02T08:49:42.129 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:49:42.130 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:49:42.130 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:49:42.130 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:49:42.131 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:49:42.131 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:18.0.0-4632.g6450d270.el8 ceph 150 M 2023-07-02T08:49:42.131 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:49:42.131 INFO:teuthology.orchestra.run.smithi164.stdout: cryptopp x86_64 8.6.0-1.el8 epel 1.3 M 2023-07-02T08:49:42.131 INFO:teuthology.orchestra.run.smithi164.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-07-02T08:49:42.131 INFO:teuthology.orchestra.run.smithi164.stdout: libasan6 x86_64 11.2.1-9.1.el8 appstream 402 k 2023-07-02T08:49:42.131 INFO:teuthology.orchestra.run.smithi164.stdout: libatomic x86_64 8.5.0-20.el8 baseos 26 k 2023-07-02T08:49:42.132 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:18.0.0-4632.g6450d270.el8 ceph 135 k 2023-07-02T08:49:42.132 INFO:teuthology.orchestra.run.smithi164.stdout: libubsan x86_64 8.5.0-20.el8 baseos 152 k 2023-07-02T08:49:42.132 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-07-02T08:49:42.132 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-07-02T08:49:42.132 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-07-02T08:49:42.132 INFO:teuthology.orchestra.run.smithi164.stdout: yaml-cpp x86_64 0.6.2-3.el8 epel 171 k 2023-07-02T08:49:42.133 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:49:42.133 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:49:42.133 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:49:42.133 INFO:teuthology.orchestra.run.smithi164.stdout:Install 11 Packages 2023-07-02T08:49:42.133 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:49:42.134 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 153 M 2023-07-02T08:49:42.134 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 1.7 G 2023-07-02T08:49:42.134 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:49:42.135 INFO:teuthology.orchestra.run.smithi103.stdout:(8/11): libubsan-8.5.0-20.el8.x86_64.rpm 520 kB/s | 152 kB 00:00 2023-07-02T08:49:42.477 INFO:teuthology.orchestra.run.smithi103.stdout:(9/11): yaml-cpp-0.6.2-3.el8.x86_64.rpm 500 kB/s | 171 kB 00:00 2023-07-02T08:49:42.752 INFO:teuthology.orchestra.run.smithi103.stdout:(10/11): cryptopp-8.6.0-1.el8.x86_64.rpm 1.4 MB/s | 1.3 MB 00:00 2023-07-02T08:49:42.924 INFO:teuthology.orchestra.run.smithi164.stdout:(1/11): jq-1.6-7.el8.x86_64.rpm 472 kB/s | 202 kB 00:00 2023-07-02T08:49:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:(2/11): libasan6-11.2.1-9.1.el8.x86_64.rpm 2.3 MB/s | 402 kB 00:00 2023-07-02T08:49:43.205 INFO:teuthology.orchestra.run.smithi164.stdout:(3/11): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-07-02T08:49:43.305 INFO:teuthology.orchestra.run.smithi164.stdout:(4/11): socat-1.7.4.1-1.el8.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2023-07-02T08:49:43.422 INFO:teuthology.orchestra.run.smithi164.stdout:(5/11): libcephsqlite-18.0.0-4632.g6450d270.el8 145 kB/s | 135 kB 00:00 2023-07-02T08:49:43.447 INFO:teuthology.orchestra.run.smithi164.stdout:(6/11): xmlstarlet-1.6.1-20.el8.x86_64.rpm 497 kB/s | 71 kB 00:00 2023-07-02T08:49:43.523 INFO:teuthology.orchestra.run.smithi164.stdout:(7/11): libatomic-8.5.0-20.el8.x86_64.rpm 255 kB/s | 26 kB 00:00 2023-07-02T08:49:43.573 INFO:teuthology.orchestra.run.smithi164.stdout:(8/11): libubsan-8.5.0-20.el8.x86_64.rpm 1.2 MB/s | 152 kB 00:00 2023-07-02T08:49:43.832 INFO:teuthology.orchestra.run.smithi164.stdout:(9/11): yaml-cpp-0.6.2-3.el8.x86_64.rpm 673 kB/s | 171 kB 00:00 2023-07-02T08:49:43.923 INFO:teuthology.orchestra.run.smithi164.stdout:(10/11): cryptopp-8.6.0-1.el8.x86_64.rpm 3.2 MB/s | 1.3 MB 00:00 2023-07-02T08:52:18.029 INFO:teuthology.orchestra.run.smithi164.stdout:(11/11): ceph-test-18.0.0-4632.g6450d270.el8.x8 989 kB/s | 150 MB 02:35 2023-07-02T08:52:18.031 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:52:18.031 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.0 MB/s | 153 MB 02:35 2023-07-02T08:52:18.090 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:52:18.112 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:52:18.112 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:52:18.836 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:52:18.839 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:52:19.883 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:52:20.199 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : yaml-cpp-0.6.2-3.el8.x86_64 1/11 2023-07-02T08:52:20.376 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cryptopp-8.6.0-1.el8.x86_64 2/11 2023-07-02T08:52:20.550 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libubsan-8.5.0-20.el8.x86_64 3/11 2023-07-02T08:52:20.616 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libubsan-8.5.0-20.el8.x86_64 3/11 2023-07-02T08:52:20.775 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libatomic-8.5.0-20.el8.x86_64 4/11 2023-07-02T08:52:20.837 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libatomic-8.5.0-20.el8.x86_64 4/11 2023-07-02T08:52:21.087 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 5/11 2023-07-02T08:52:21.295 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 6/11 2023-07-02T08:52:21.442 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 7/11 2023-07-02T08:52:21.519 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 7/11 2023-07-02T08:52:21.725 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : jq-1.6-7.el8.x86_64 8/11 2023-07-02T08:52:21.892 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-07-02T08:52:21.965 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-07-02T08:52:22.150 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:18.0.0-4632.g6450d270.el8.x86_64 10/11 2023-07-02T08:52:31.160 INFO:teuthology.orchestra.run.smithi103.stdout:(11/11): ceph-test-18.0.0-4632.g6450d270.el8.x8 902 kB/s | 150 MB 02:50 2023-07-02T08:52:31.163 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:52:31.163 INFO:teuthology.orchestra.run.smithi103.stdout:Total 918 kB/s | 153 MB 02:50 2023-07-02T08:52:31.222 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:52:31.245 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:52:31.245 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:52:31.951 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:52:31.954 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:52:32.935 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:52:33.231 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : yaml-cpp-0.6.2-3.el8.x86_64 1/11 2023-07-02T08:52:33.427 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cryptopp-8.6.0-1.el8.x86_64 2/11 2023-07-02T08:52:33.548 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libubsan-8.5.0-20.el8.x86_64 3/11 2023-07-02T08:52:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libubsan-8.5.0-20.el8.x86_64 3/11 2023-07-02T08:52:33.740 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libatomic-8.5.0-20.el8.x86_64 4/11 2023-07-02T08:52:33.810 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libatomic-8.5.0-20.el8.x86_64 4/11 2023-07-02T08:52:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 5/11 2023-07-02T08:52:34.268 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 6/11 2023-07-02T08:52:34.406 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 7/11 2023-07-02T08:52:34.485 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 7/11 2023-07-02T08:52:34.698 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : jq-1.6-7.el8.x86_64 8/11 2023-07-02T08:52:34.865 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-07-02T08:52:34.923 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-07-02T08:52:35.065 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:18.0.0-4632.g6450d270.el8.x86_64 10/11 2023-07-02T08:52:45.378 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4632.g6450d270.el8.x86_64 10/11 2023-07-02T08:52:45.628 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:18.0.0-4632.g6450d270.el8.x86_64 11/11 2023-07-02T08:52:45.955 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-test-2:18.0.0-4632.g6450d270.el8.x86_64 11/11 2023-07-02T08:52:45.956 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:18.0.0-4632.g6450d270.el8.x86_64 1/11 2023-07-02T08:52:45.956 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:18.0.0-4632.g6450d270.el8.x86_64 2/11 2023-07-02T08:52:45.956 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : jq-1.6-7.el8.x86_64 3/11 2023-07-02T08:52:45.956 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libasan6-11.2.1-9.1.el8.x86_64 4/11 2023-07-02T08:52:45.956 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/11 2023-07-02T08:52:45.956 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 6/11 2023-07-02T08:52:45.957 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 7/11 2023-07-02T08:52:45.957 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libatomic-8.5.0-20.el8.x86_64 8/11 2023-07-02T08:52:45.957 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libubsan-8.5.0-20.el8.x86_64 9/11 2023-07-02T08:52:45.957 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cryptopp-8.6.0-1.el8.x86_64 10/11 2023-07-02T08:52:46.627 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : yaml-cpp-0.6.2-3.el8.x86_64 11/11 2023-07-02T08:52:46.627 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:52:46.627 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi164.stdout: cryptopp-8.6.0-1.el8.x86_64 2023-07-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi164.stdout: jq-1.6-7.el8.x86_64 2023-07-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi164.stdout: libasan6-11.2.1-9.1.el8.x86_64 2023-07-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi164.stdout: libatomic-8.5.0-20.el8.x86_64 2023-07-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi164.stdout: libubsan-8.5.0-20.el8.x86_64 2023-07-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi164.stdout: yaml-cpp-0.6.2-3.el8.x86_64 2023-07-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:52:46.945 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph 2023-07-02T08:52:47.782 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:04:26 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:52:49.110 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:52:49.114 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:52:49.114 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:52:49.114 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:52:49.114 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:52:49.114 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:18.0.0-4632.g6450d270.el8 ceph 6.4 k 2023-07-02T08:52:49.115 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:52:49.115 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:18.0.0-4632.g6450d270.el8 ceph 2.2 M 2023-07-02T08:52:49.115 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:18.0.0-4632.g6450d270.el8 ceph 1.5 M 2023-07-02T08:52:49.115 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 240 k 2023-07-02T08:52:49.115 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:18.0.0-4632.g6450d270.el8 ceph 9.0 M 2023-07-02T08:52:49.115 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:18.0.0-4632.g6450d270.el8 ceph 160 M 2023-07-02T08:52:49.115 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-02T08:52:49.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-02T08:52:49.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-07-02T08:52:49.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-02T08:52:49.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-02T08:52:49.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-07-02T08:52:49.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-02T08:52:49.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-02T08:52:49.117 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-02T08:52:49.117 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-07-02T08:52:49.117 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-07-02T08:52:49.117 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-02T08:52:49.117 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-02T08:52:49.117 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-02T08:52:49.117 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-07-02T08:52:49.118 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-07-02T08:52:49.118 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2017.2-9.el8 appstream 54 k 2023-07-02T08:52:49.118 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-02T08:52:49.118 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-02T08:52:49.118 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-02T08:52:49.118 INFO:teuthology.orchestra.run.smithi164.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-02T08:52:49.118 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-02T08:52:49.118 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-02T08:52:49.119 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-02T08:52:49.119 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-07-02T08:52:49.119 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-02T08:52:49.119 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:52:49.119 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:52:49.119 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:52:49.119 INFO:teuthology.orchestra.run.smithi164.stdout:Install 32 Packages 2023-07-02T08:52:49.119 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:52:49.120 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 177 M 2023-07-02T08:52:49.120 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 2.3 G 2023-07-02T08:52:49.120 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:52:49.899 INFO:teuthology.orchestra.run.smithi164.stdout:(1/32): ceph-18.0.0-4632.g6450d270.el8.x86_64.r 17 kB/s | 6.4 kB 00:00 2023-07-02T08:52:51.798 INFO:teuthology.orchestra.run.smithi164.stdout:(2/32): ceph-mgr-18.0.0-4632.g6450d270.el8.x86_ 674 kB/s | 1.5 MB 00:02 2023-07-02T08:52:56.449 INFO:teuthology.orchestra.run.smithi164.stdout:(3/32): ceph-mon-18.0.0-4632.g6450d270.el8.x86_ 1.4 MB/s | 9.0 MB 00:06 2023-07-02T08:52:56.641 INFO:teuthology.orchestra.run.smithi164.stdout:(4/32): ceph-mgr-modules-core-18.0.0-4632.g6450 1.2 MB/s | 240 kB 00:00 2023-07-02T08:52:56.899 INFO:teuthology.orchestra.run.smithi164.stdout:(5/32): ceph-mds-18.0.0-4632.g6450d270.el8.x86_ 306 kB/s | 2.2 MB 00:07 2023-07-02T08:52:57.025 INFO:teuthology.orchestra.run.smithi164.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 409 kB/s | 157 kB 00:00 2023-07-02T08:52:57.142 INFO:teuthology.orchestra.run.smithi164.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 887 kB/s | 103 kB 00:00 2023-07-02T08:52:57.167 INFO:teuthology.orchestra.run.smithi164.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 145 kB/s | 39 kB 00:00 2023-07-02T08:52:57.451 INFO:teuthology.orchestra.run.smithi164.stdout:(9/32): python3-pytz-2017.2-9.el8.noarch.rpm 175 kB/s | 54 kB 00:00 2023-07-02T08:52:57.993 INFO:teuthology.orchestra.run.smithi164.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 554 kB/s | 457 kB 00:00 2023-07-02T08:52:58.288 INFO:teuthology.orchestra.run.smithi164.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 297 kB/s | 249 kB 00:00 2023-07-02T08:52:58.351 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4632.g6450d270.el8.x86_64 10/11 2023-07-02T08:52:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:18.0.0-4632.g6450d270.el8.x86_64 11/11 2023-07-02T08:52:58.710 INFO:teuthology.orchestra.run.smithi164.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 259 kB/s | 109 kB 00:00 2023-07-02T08:52:58.888 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:18.0.0-4632.g6450d270.el8.x86_64 11/11 2023-07-02T08:52:58.888 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:18.0.0-4632.g6450d270.el8.x86_64 1/11 2023-07-02T08:52:58.889 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:18.0.0-4632.g6450d270.el8.x86_64 2/11 2023-07-02T08:52:58.889 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : jq-1.6-7.el8.x86_64 3/11 2023-07-02T08:52:58.889 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libasan6-11.2.1-9.1.el8.x86_64 4/11 2023-07-02T08:52:58.889 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/11 2023-07-02T08:52:58.889 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 6/11 2023-07-02T08:52:58.889 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 7/11 2023-07-02T08:52:58.889 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libatomic-8.5.0-20.el8.x86_64 8/11 2023-07-02T08:52:58.890 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libubsan-8.5.0-20.el8.x86_64 9/11 2023-07-02T08:52:58.890 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cryptopp-8.6.0-1.el8.x86_64 10/11 2023-07-02T08:52:59.044 INFO:teuthology.orchestra.run.smithi164.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 610 kB/s | 641 kB 00:01 2023-07-02T08:52:59.433 INFO:teuthology.orchestra.run.smithi164.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 60 kB/s | 44 kB 00:00 2023-07-02T08:52:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : yaml-cpp-0.6.2-3.el8.x86_64 11/11 2023-07-02T08:52:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:52:59.591 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:52:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:52:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: cryptopp-8.6.0-1.el8.x86_64 2023-07-02T08:52:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: jq-1.6-7.el8.x86_64 2023-07-02T08:52:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: libasan6-11.2.1-9.1.el8.x86_64 2023-07-02T08:52:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: libatomic-8.5.0-20.el8.x86_64 2023-07-02T08:52:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:52:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: libubsan-8.5.0-20.el8.x86_64 2023-07-02T08:52:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-02T08:52:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-02T08:52:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-02T08:52:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: yaml-cpp-0.6.2-3.el8.x86_64 2023-07-02T08:52:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:52:59.593 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:52:59.733 INFO:teuthology.orchestra.run.smithi164.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 269 kB/s | 185 kB 00:00 2023-07-02T08:52:59.894 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2023-07-02T08:53:00.177 INFO:teuthology.orchestra.run.smithi164.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 867 kB/s | 384 kB 00:00 2023-07-02T08:53:00.420 INFO:teuthology.orchestra.run.smithi164.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 175 kB/s | 173 kB 00:00 2023-07-02T08:53:00.725 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:04:39 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:53:00.729 INFO:teuthology.orchestra.run.smithi164.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 20 kB/s | 11 kB 00:00 2023-07-02T08:53:01.046 INFO:teuthology.orchestra.run.smithi164.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 28 kB/s | 18 kB 00:00 2023-07-02T08:53:01.371 INFO:teuthology.orchestra.run.smithi164.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 76 kB/s | 49 kB 00:00 2023-07-02T08:53:01.705 INFO:teuthology.orchestra.run.smithi164.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 89 kB/s | 59 kB 00:00 2023-07-02T08:53:01.956 INFO:teuthology.orchestra.run.smithi164.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 484 kB/s | 283 kB 00:00 2023-07-02T08:53:02.060 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:53:02.063 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:53:02.063 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:53:02.063 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:53:02.064 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:53:02.064 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:18.0.0-4632.g6450d270.el8 ceph 6.4 k 2023-07-02T08:53:02.064 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:53:02.064 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:18.0.0-4632.g6450d270.el8 ceph 2.2 M 2023-07-02T08:53:02.064 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:18.0.0-4632.g6450d270.el8 ceph 1.5 M 2023-07-02T08:53:02.064 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 240 k 2023-07-02T08:53:02.064 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:18.0.0-4632.g6450d270.el8 ceph 9.0 M 2023-07-02T08:53:02.065 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:18.0.0-4632.g6450d270.el8 ceph 160 M 2023-07-02T08:53:02.065 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-02T08:53:02.065 INFO:teuthology.orchestra.run.smithi103.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-02T08:53:02.065 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-07-02T08:53:02.065 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-02T08:53:02.065 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-02T08:53:02.065 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-07-02T08:53:02.066 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-02T08:53:02.066 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-02T08:53:02.066 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-02T08:53:02.066 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-07-02T08:53:02.066 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-07-02T08:53:02.066 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-02T08:53:02.066 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-02T08:53:02.066 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-02T08:53:02.067 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-07-02T08:53:02.067 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-07-02T08:53:02.067 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2017.2-9.el8 appstream 54 k 2023-07-02T08:53:02.067 INFO:teuthology.orchestra.run.smithi103.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-02T08:53:02.067 INFO:teuthology.orchestra.run.smithi103.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-02T08:53:02.067 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-02T08:53:02.067 INFO:teuthology.orchestra.run.smithi103.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-02T08:53:02.068 INFO:teuthology.orchestra.run.smithi103.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-02T08:53:02.068 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-02T08:53:02.068 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-02T08:53:02.068 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-07-02T08:53:02.068 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-02T08:53:02.068 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:53:02.068 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:53:02.068 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:53:02.069 INFO:teuthology.orchestra.run.smithi103.stdout:Install 32 Packages 2023-07-02T08:53:02.069 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:53:02.069 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 177 M 2023-07-02T08:53:02.069 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.3 G 2023-07-02T08:53:02.070 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:53:02.272 INFO:teuthology.orchestra.run.smithi164.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 27 kB/s | 16 kB 00:00 2023-07-02T08:53:02.589 INFO:teuthology.orchestra.run.smithi164.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 30 kB/s | 19 kB 00:00 2023-07-02T08:53:02.899 INFO:teuthology.orchestra.run.smithi164.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 38 kB/s | 24 kB 00:00 2023-07-02T08:53:02.967 INFO:teuthology.orchestra.run.smithi103.stdout:(1/32): ceph-18.0.0-4632.g6450d270.el8.x86_64.r 15 kB/s | 6.4 kB 00:00 2023-07-02T08:53:03.224 INFO:teuthology.orchestra.run.smithi164.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 46 kB/s | 29 kB 00:00 2023-07-02T08:53:03.483 INFO:teuthology.orchestra.run.smithi164.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 47 kB/s | 27 kB 00:00 2023-07-02T08:53:03.749 INFO:teuthology.orchestra.run.smithi164.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 486 kB/s | 255 kB 00:00 2023-07-02T08:53:04.067 INFO:teuthology.orchestra.run.smithi164.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 430 kB/s | 251 kB 00:00 2023-07-02T08:53:04.400 INFO:teuthology.orchestra.run.smithi164.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 131 kB/s | 85 kB 00:00 2023-07-02T08:53:04.734 INFO:teuthology.orchestra.run.smithi164.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 34 kB/s | 23 kB 00:00 2023-07-02T08:53:04.797 INFO:teuthology.orchestra.run.smithi103.stdout:(2/32): ceph-mgr-18.0.0-4632.g6450d270.el8.x86_ 675 kB/s | 1.5 MB 00:02 2023-07-02T08:53:05.426 INFO:teuthology.orchestra.run.smithi103.stdout:(3/32): ceph-mds-18.0.0-4632.g6450d270.el8.x86_ 779 kB/s | 2.2 MB 00:02 2023-07-02T08:53:05.768 INFO:teuthology.orchestra.run.smithi103.stdout:(4/32): ceph-mgr-modules-core-18.0.0-4632.g6450 704 kB/s | 240 kB 00:00 2023-07-02T08:53:06.152 INFO:teuthology.orchestra.run.smithi103.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 410 kB/s | 157 kB 00:00 2023-07-02T08:53:06.244 INFO:teuthology.orchestra.run.smithi103.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 423 kB/s | 39 kB 00:00 2023-07-02T08:53:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2023-07-02T08:53:06.478 INFO:teuthology.orchestra.run.smithi103.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 406 kB/s | 54 kB 00:00 2023-07-02T08:53:06.653 INFO:teuthology.orchestra.run.smithi103.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 2.6 MB/s | 457 kB 00:00 2023-07-02T08:53:06.862 INFO:teuthology.orchestra.run.smithi103.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2023-07-02T08:53:06.962 INFO:teuthology.orchestra.run.smithi103.stdout:(11/32): python3-cryptography-3.2.1-6.el8.x86_6 6.3 MB/s | 641 kB 00:00 2023-07-02T08:53:07.021 INFO:teuthology.orchestra.run.smithi103.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2023-07-02T08:53:07.113 INFO:teuthology.orchestra.run.smithi103.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 475 kB/s | 44 kB 00:00 2023-07-02T08:53:07.188 INFO:teuthology.orchestra.run.smithi103.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.4 MB/s | 185 kB 00:00 2023-07-02T08:53:07.238 INFO:teuthology.orchestra.run.smithi103.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 3.4 MB/s | 173 kB 00:00 2023-07-02T08:53:07.314 INFO:teuthology.orchestra.run.smithi103.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 5.0 MB/s | 384 kB 00:00 2023-07-02T08:53:07.356 INFO:teuthology.orchestra.run.smithi103.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 266 kB/s | 11 kB 00:00 2023-07-02T08:53:07.397 INFO:teuthology.orchestra.run.smithi103.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 423 kB/s | 18 kB 00:00 2023-07-02T08:53:07.440 INFO:teuthology.orchestra.run.smithi103.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 1.1 MB/s | 49 kB 00:00 2023-07-02T08:53:07.481 INFO:teuthology.orchestra.run.smithi103.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.4 MB/s | 59 kB 00:00 2023-07-02T08:53:07.524 INFO:teuthology.orchestra.run.smithi103.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 6.6 MB/s | 283 kB 00:00 2023-07-02T08:53:07.568 INFO:teuthology.orchestra.run.smithi103.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 357 kB/s | 16 kB 00:00 2023-07-02T08:53:07.609 INFO:teuthology.orchestra.run.smithi103.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 450 kB/s | 19 kB 00:00 2023-07-02T08:53:07.651 INFO:teuthology.orchestra.run.smithi103.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 574 kB/s | 24 kB 00:00 2023-07-02T08:53:07.694 INFO:teuthology.orchestra.run.smithi103.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 702 kB/s | 29 kB 00:00 2023-07-02T08:53:07.735 INFO:teuthology.orchestra.run.smithi103.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 663 kB/s | 27 kB 00:00 2023-07-02T08:53:07.777 INFO:teuthology.orchestra.run.smithi103.stdout:(27/32): python3-waitress-1.4.3-1.el8.noarch.rp 6.0 MB/s | 255 kB 00:00 2023-07-02T08:53:07.819 INFO:teuthology.orchestra.run.smithi103.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 5.9 MB/s | 251 kB 00:00 2023-07-02T08:53:07.861 INFO:teuthology.orchestra.run.smithi103.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.0 MB/s | 85 kB 00:00 2023-07-02T08:53:07.895 INFO:teuthology.orchestra.run.smithi103.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 691 kB/s | 23 kB 00:00 2023-07-02T08:53:13.379 INFO:teuthology.orchestra.run.smithi103.stdout:(31/32): ceph-mon-18.0.0-4632.g6450d270.el8.x86 883 kB/s | 9.0 MB 00:10 2023-07-02T08:55:39.380 INFO:teuthology.orchestra.run.smithi164.stdout:(32/32): ceph-osd-18.0.0-4632.g6450d270.el8.x86 978 kB/s | 160 MB 02:47 2023-07-02T08:55:39.382 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:55:39.383 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.0 MB/s | 177 MB 02:50 2023-07-02T08:55:39.452 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:55:39.491 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:55:39.491 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:55:40.341 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:55:40.344 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:55:42.157 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:55:42.358 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-07-02T08:55:42.609 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-07-02T08:55:42.818 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-07-02T08:55:43.083 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-07-02T08:55:43.311 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-07-02T08:55:43.495 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-07-02T08:55:43.721 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-07-02T08:55:43.907 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-07-02T08:55:44.192 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-07-02T08:55:44.437 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-07-02T08:55:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-07-02T08:55:44.982 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-07-02T08:55:45.366 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-07-02T08:55:45.671 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-07-02T08:55:45.884 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-07-02T08:55:46.085 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-07-02T08:55:46.285 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-07-02T08:55:46.678 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-07-02T08:55:46.915 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-07-02T08:55:47.128 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-07-02T08:55:47.347 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-07-02T08:55:47.606 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-07-02T08:55:47.913 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-07-02T08:55:48.145 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-07-02T08:55:48.463 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-07-02T08:55:48.832 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-07-02T08:55:49.227 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4632.g6450d270.el8. 27/32 2023-07-02T08:55:49.420 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 28/32 2023-07-02T08:56:17.665 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 28/32 2023-07-02T08:56:17.899 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:18.0.0-4632.g6450d270.el8.x86_64 29/32 2023-07-02T08:56:18.805 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:18.0.0-4632.g6450d270.el8.x86_64 29/32 2023-07-02T08:56:19.021 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:18.0.0-4632.g6450d270.el8.x86_64 30/32 2023-07-02T08:56:19.238 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:18.0.0-4632.g6450d270.el8.x86_64 30/32 2023-07-02T08:56:19.438 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:18.0.0-4632.g6450d270.el8.x86_64 31/32 2023-07-02T08:56:19.454 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:18.0.0-4632.g6450d270.el8.x86_64 31/32 2023-07-02T08:56:19.562 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:18.0.0-4632.g6450d270.el8.x86_64 32/32 2023-07-02T08:56:19.970 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-2:18.0.0-4632.g6450d270.el8.x86_64 32/32 2023-07-02T08:56:19.971 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:18.0.0-4632.g6450d270.el8.x86_64 1/32 2023-07-02T08:56:19.971 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:18.0.0-4632.g6450d270.el8.x86_64 2/32 2023-07-02T08:56:19.971 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 3/32 2023-07-02T08:56:19.971 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:18.0.0-4632.g6450d270.el8.x86_64 4/32 2023-07-02T08:56:19.971 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:18.0.0-4632.g6450d270.el8.x86_64 5/32 2023-07-02T08:56:19.971 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4632.g6450d270.el8. 6/32 2023-07-02T08:56:19.971 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-07-02T08:56:19.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-07-02T08:56:19.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-07-02T08:56:19.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-07-02T08:56:19.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-07-02T08:56:19.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-07-02T08:56:19.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-07-02T08:56:19.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-07-02T08:56:19.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-07-02T08:56:19.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-07-02T08:56:19.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-07-02T08:56:19.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-07-02T08:56:19.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-07-02T08:56:19.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-07-02T08:56:19.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-07-02T08:56:19.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-07-02T08:56:19.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-07-02T08:56:19.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-07-02T08:56:19.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-07-02T08:56:19.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-07-02T08:56:19.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-07-02T08:56:19.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-07-02T08:56:19.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-07-02T08:56:19.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-07-02T08:56:19.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-07-02T08:56:21.124 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-07-02T08:56:21.125 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:21.125 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:56:21.125 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:56:21.125 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:56:21.125 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:56:21.126 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:56:21.126 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:56:21.126 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:56:21.126 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-02T08:56:21.126 INFO:teuthology.orchestra.run.smithi164.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-02T08:56:21.126 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-07-02T08:56:21.126 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-02T08:56:21.126 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-02T08:56:21.127 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-07-02T08:56:21.127 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-02T08:56:21.127 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-02T08:56:21.127 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-02T08:56:21.127 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.0.6-14.el8.noarch 2023-07-02T08:56:21.127 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-02T08:56:21.127 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-02T08:56:21.128 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-02T08:56:21.128 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-2.6-1.el8.noarch 2023-07-02T08:56:21.128 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-02T08:56:21.128 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-02T08:56:21.128 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-02T08:56:21.128 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-02T08:56:21.128 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-02T08:56:21.129 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-02T08:56:21.129 INFO:teuthology.orchestra.run.smithi164.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-02T08:56:21.129 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-02T08:56:21.129 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-02T08:56:21.129 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-02T08:56:21.129 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-02T08:56:21.129 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-02T08:56:21.129 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:21.130 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:56:23.408 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-base 2023-07-02T08:56:24.252 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:08:03 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:56:25.413 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:56:25.566 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:56:25.567 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:56:25.567 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:56:25.876 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install cephadm 2023-07-02T08:56:26.718 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:08:05 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:56:28.056 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:56:28.057 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:28.057 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:56:28.057 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:28.057 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:56:28.057 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 164 k 2023-07-02T08:56:28.058 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:28.058 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:56:28.058 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:28.058 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2023-07-02T08:56:28.058 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:28.059 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 164 k 2023-07-02T08:56:28.059 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 654 k 2023-07-02T08:56:28.059 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:56:28.988 INFO:teuthology.orchestra.run.smithi164.stdout:cephadm-18.0.0-4632.g6450d270.el8.noarch.rpm 176 kB/s | 164 kB 00:00 2023-07-02T08:56:28.988 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:56:28.989 INFO:teuthology.orchestra.run.smithi164.stdout:Total 176 kB/s | 164 kB 00:00 2023-07-02T08:56:28.989 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:56:28.998 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:56:28.998 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:56:29.001 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:56:29.004 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:56:29.555 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:56:31.442 INFO:teuthology.orchestra.run.smithi103.stdout:(32/32): ceph-osd-18.0.0-4632.g6450d270.el8.x86 793 kB/s | 160 MB 03:26 2023-07-02T08:56:31.445 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:56:31.445 INFO:teuthology.orchestra.run.smithi103.stdout:Total 864 kB/s | 177 MB 03:29 2023-07-02T08:56:31.520 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:56:31.559 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:56:31.560 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:56:32.409 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:56:32.412 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:56:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:56:34.272 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-07-02T08:56:34.541 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-07-02T08:56:34.766 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-07-02T08:56:34.982 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-07-02T08:56:35.162 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-07-02T08:56:35.401 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-07-02T08:56:35.578 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-07-02T08:56:35.713 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-07-02T08:56:35.895 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-07-02T08:56:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:56:36.157 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-07-02T08:56:36.362 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-07-02T08:56:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-07-02T08:56:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-07-02T08:56:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-07-02T08:56:37.349 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-07-02T08:56:37.557 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-07-02T08:56:37.775 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-07-02T08:56:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:56:38.081 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-07-02T08:56:38.146 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:56:38.288 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-07-02T08:56:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-07-02T08:56:38.679 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-07-02T08:56:38.946 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-07-02T08:56:39.178 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-07-02T08:56:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-07-02T08:56:39.429 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:56:39.430 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:39.430 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:56:39.430 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:56:39.430 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:39.430 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:56:39.660 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-immutable-object-cache 2023-07-02T08:56:39.710 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-07-02T08:56:40.029 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-07-02T08:56:40.434 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4632.g6450d270.el8. 27/32 2023-07-02T08:56:40.524 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:08:19 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:56:40.619 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 28/32 2023-07-02T08:56:41.847 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:56:41.848 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2023-07-02T08:56:41.848 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:41.848 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:56:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4632.g6450d270.el8 ceph 157 k 2023-07-02T08:56:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:41.849 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:56:41.849 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:41.849 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2023-07-02T08:56:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:41.850 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 157 k 2023-07-02T08:56:41.850 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 540 k 2023-07-02T08:56:41.850 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:56:42.367 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-immutable-object-cache-18.0.0-4632.g6450d2 302 kB/s | 157 kB 00:00 2023-07-02T08:56:42.368 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:56:42.368 INFO:teuthology.orchestra.run.smithi164.stdout:Total 302 kB/s | 157 kB 00:00 2023-07-02T08:56:42.369 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:56:42.379 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:56:42.379 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:56:42.427 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:56:42.430 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:56:42.570 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:56:42.792 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4632.g6450d270. 1/1 2023-07-02T08:56:43.209 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4632.g6450d270. 1/1 2023-07-02T08:56:43.796 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4632.g6450d270. 1/1 2023-07-02T08:56:43.796 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:43.797 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:56:43.797 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:56:43.797 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:43.797 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:56:44.054 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr 2023-07-02T08:56:44.890 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:08:23 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:56:46.063 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:56:46.214 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:56:46.216 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:56:46.216 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:56:46.309 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-dashboard 2023-07-02T08:56:47.136 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:08:26 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:56:48.452 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:56:48.453 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:48.453 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:56:48.454 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:48.454 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:56:48.454 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 1.7 M 2023-07-02T08:56:48.454 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:56:48.454 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 23 k 2023-07-02T08:56:48.454 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 15 k 2023-07-02T08:56:48.454 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-07-02T08:56:48.455 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-02T08:56:48.455 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-02T08:56:48.455 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:48.455 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:56:48.455 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:56:48.455 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2023-07-02T08:56:48.455 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:48.456 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 2.0 M 2023-07-02T08:56:48.456 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 8.2 M 2023-07-02T08:56:48.456 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:56:48.962 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): ceph-grafana-dashboards-18.0.0-4632.g645 53 kB/s | 23 kB 00:00 2023-07-02T08:56:49.013 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): ceph-prometheus-alerts-18.0.0-4632.g6450 30 kB/s | 15 kB 00:00 2023-07-02T08:56:49.055 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 462 kB/s | 43 kB 00:00 2023-07-02T08:56:49.272 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 130 kB/s | 34 kB 00:00 2023-07-02T08:56:49.455 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 491 kB/s | 196 kB 00:00 2023-07-02T08:56:50.373 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): ceph-mgr-dashboard-18.0.0-4632.g6450d270 949 kB/s | 1.7 MB 00:01 2023-07-02T08:56:50.375 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:56:50.375 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.0 MB/s | 2.0 MB 00:01 2023-07-02T08:56:50.386 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:56:50.396 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:56:50.396 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:56:50.491 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:56:50.495 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:56:50.788 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:56:51.018 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-07-02T08:56:51.287 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-07-02T08:56:51.510 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-07-02T08:56:51.622 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4632.g6450d270.el8.n 4/6 2023-07-02T08:56:52.010 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4632.g6450d270.el8. 5/6 2023-07-02T08:56:52.252 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4632.g6450d270.el8.noarc 6/6 2023-07-02T08:56:52.521 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4632.g6450d270.el8.noarc 6/6 2023-07-02T08:56:52.521 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4632.g6450d270.el8. 1/6 2023-07-02T08:56:52.522 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4632.g6450d270.el8.noarc 2/6 2023-07-02T08:56:52.522 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4632.g6450d270.el8.n 3/6 2023-07-02T08:56:52.522 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-07-02T08:56:52.522 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-07-02T08:56:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-07-02T08:56:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:53.087 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:56:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:56:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:56:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:56:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-02T08:56:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-02T08:56:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-02T08:56:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:53.088 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:56:53.226 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-diskprediction-local 2023-07-02T08:56:54.058 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:08:33 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:56:55.396 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:56:55.397 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================================ 2023-07-02T08:56:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:56:55.398 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================================ 2023-07-02T08:56:55.398 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:56:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 7.4 M 2023-07-02T08:56:55.398 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:56:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-07-02T08:56:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-07-02T08:56:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-07-02T08:56:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-07-02T08:56:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-07-02T08:56:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-07-02T08:56:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-07-02T08:56:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:55.399 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:56:55.400 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================================ 2023-07-02T08:56:55.400 INFO:teuthology.orchestra.run.smithi164.stdout:Install 8 Packages 2023-07-02T08:56:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:56:55.400 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 35 M 2023-07-02T08:56:55.401 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 210 M 2023-07-02T08:56:55.401 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:56:56.353 INFO:teuthology.orchestra.run.smithi164.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 5.5 MB/s | 4.7 MB 00:00 2023-07-02T08:56:56.420 INFO:teuthology.orchestra.run.smithi164.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.2 MB/s | 4.9 MB 00:00 2023-07-02T08:56:56.520 INFO:teuthology.orchestra.run.smithi164.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2023-07-02T08:56:56.679 INFO:teuthology.orchestra.run.smithi164.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2023-07-02T08:56:56.838 INFO:teuthology.orchestra.run.smithi164.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 645 kB 00:00 2023-07-02T08:56:56.880 INFO:teuthology.orchestra.run.smithi164.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 4.1 MB/s | 172 kB 00:00 2023-07-02T08:56:57.163 INFO:teuthology.orchestra.run.smithi164.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 21 MB/s | 14 MB 00:00 2023-07-02T08:56:58.728 INFO:teuthology.orchestra.run.smithi164.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-463 2.3 MB/s | 7.4 MB 00:03 2023-07-02T08:56:58.729 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:56:58.729 INFO:teuthology.orchestra.run.smithi164.stdout:Total 11 MB/s | 35 MB 00:03 2023-07-02T08:56:58.965 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:56:58.983 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:56:58.984 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:56:59.360 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:56:59.363 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:56:59.958 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:57:00.106 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-07-02T08:57:00.268 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-07-02T08:57:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-07-02T08:57:01.046 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-07-02T08:57:01.264 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-07-02T08:57:01.873 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-07-02T08:57:02.022 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-07-02T08:57:02.718 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-07-02T08:57:03.059 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-07-02T08:57:03.407 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-02T08:57:05.205 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-02T08:57:06.542 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-07-02T08:57:06.835 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4632.g6450d27 8/8 2023-07-02T08:57:07.160 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4632.g6450d27 8/8 2023-07-02T08:57:07.161 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4632.g6450d27 1/8 2023-07-02T08:57:07.161 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-07-02T08:57:07.161 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-07-02T08:57:07.161 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-07-02T08:57:07.161 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-07-02T08:57:07.161 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-07-02T08:57:07.162 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-07-02T08:57:07.805 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-07-02T08:57:07.805 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:07.805 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:57:07.806 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:07.806 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-07-02T08:57:07.806 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-07-02T08:57:07.806 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.15-4.el8.x86_64 2023-07-02T08:57:07.806 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-07-02T08:57:07.806 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-02T08:57:07.806 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-02T08:57:07.807 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-07-02T08:57:07.807 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:07.807 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:07.922 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-rook 2023-07-02T08:57:08.754 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:08:47 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:08.885 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 28/32 2023-07-02T08:57:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:18.0.0-4632.g6450d270.el8.x86_64 29/32 2023-07-02T08:57:09.997 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:18.0.0-4632.g6450d270.el8.x86_64 29/32 2023-07-02T08:57:10.076 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:10.078 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:10.078 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:57:10.079 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:10.079 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:57:10.079 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 47 k 2023-07-02T08:57:10.079 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:57:10.079 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-02T08:57:10.079 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-02T08:57:10.079 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-02T08:57:10.080 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-07-02T08:57:10.080 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-07-02T08:57:10.080 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-02T08:57:10.080 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-07-02T08:57:10.080 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-07-02T08:57:10.080 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-07-02T08:57:10.080 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-07-02T08:57:10.080 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-02T08:57:10.081 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-02T08:57:10.081 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:10.081 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:57:10.081 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:10.081 INFO:teuthology.orchestra.run.smithi164.stdout:Install 13 Packages 2023-07-02T08:57:10.081 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:10.082 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 1.9 M 2023-07-02T08:57:10.082 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 25 M 2023-07-02T08:57:10.082 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:57:10.213 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:18.0.0-4632.g6450d270.el8.x86_64 30/32 2023-07-02T08:57:10.430 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:18.0.0-4632.g6450d270.el8.x86_64 30/32 2023-07-02T08:57:10.596 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:18.0.0-4632.g6450d270.el8.x86_64 31/32 2023-07-02T08:57:10.613 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:18.0.0-4632.g6450d270.el8.x86_64 31/32 2023-07-02T08:57:10.728 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:18.0.0-4632.g6450d270.el8.x86_64 32/32 2023-07-02T08:57:11.117 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:18.0.0-4632.g6450d270.el8.x86_64 32/32 2023-07-02T08:57:11.118 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:18.0.0-4632.g6450d270.el8.x86_64 1/32 2023-07-02T08:57:11.118 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:18.0.0-4632.g6450d270.el8.x86_64 2/32 2023-07-02T08:57:11.118 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 3/32 2023-07-02T08:57:11.118 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:18.0.0-4632.g6450d270.el8.x86_64 4/32 2023-07-02T08:57:11.118 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:18.0.0-4632.g6450d270.el8.x86_64 5/32 2023-07-02T08:57:11.118 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4632.g6450d270.el8. 6/32 2023-07-02T08:57:11.119 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-07-02T08:57:11.119 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-07-02T08:57:11.119 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-07-02T08:57:11.119 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-07-02T08:57:11.119 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-07-02T08:57:11.119 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-07-02T08:57:11.119 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-07-02T08:57:11.120 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-07-02T08:57:11.120 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-07-02T08:57:11.120 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-07-02T08:57:11.120 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-07-02T08:57:11.120 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-07-02T08:57:11.120 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-07-02T08:57:11.120 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-07-02T08:57:11.120 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-07-02T08:57:11.121 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-07-02T08:57:11.121 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-07-02T08:57:11.121 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-07-02T08:57:11.121 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-07-02T08:57:11.122 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-07-02T08:57:11.122 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-07-02T08:57:11.122 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-07-02T08:57:11.122 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-07-02T08:57:11.122 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-07-02T08:57:11.122 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-07-02T08:57:12.054 INFO:teuthology.orchestra.run.smithi164.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 69 kB/s | 20 kB 00:00 2023-07-02T08:57:12.088 INFO:teuthology.orchestra.run.smithi164.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 82 kB/s | 27 kB 00:00 2023-07-02T08:57:12.194 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-07-02T08:57:12.194 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:12.194 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:57:12.194 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:57:12.194 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:57:12.194 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:57:12.195 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:12.195 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:57:12.195 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:57:12.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-02T08:57:12.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-02T08:57:12.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-07-02T08:57:12.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-02T08:57:12.196 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-02T08:57:12.196 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-07-02T08:57:12.196 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-02T08:57:12.196 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-02T08:57:12.196 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-02T08:57:12.196 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.0.6-14.el8.noarch 2023-07-02T08:57:12.196 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-02T08:57:12.196 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-02T08:57:12.197 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-02T08:57:12.197 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-2.6-1.el8.noarch 2023-07-02T08:57:12.197 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-02T08:57:12.197 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-02T08:57:12.197 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-02T08:57:12.197 INFO:teuthology.orchestra.run.smithi103.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-02T08:57:12.197 INFO:teuthology.orchestra.run.smithi103.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-02T08:57:12.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-02T08:57:12.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-02T08:57:12.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-02T08:57:12.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-02T08:57:12.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-02T08:57:12.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-02T08:57:12.198 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-02T08:57:12.198 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:12.199 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:57:12.313 INFO:teuthology.orchestra.run.smithi164.stdout:(3/13): ceph-mgr-rook-18.0.0-4632.g6450d270.el8 86 kB/s | 47 kB 00:00 2023-07-02T08:57:12.338 INFO:teuthology.orchestra.run.smithi164.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 445 kB/s | 126 kB 00:00 2023-07-02T08:57:12.364 INFO:teuthology.orchestra.run.smithi164.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 399 kB/s | 110 kB 00:00 2023-07-02T08:57:12.572 INFO:teuthology.orchestra.run.smithi164.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 158 kB/s | 33 kB 00:00 2023-07-02T08:57:12.639 INFO:teuthology.orchestra.run.smithi164.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 244 kB/s | 16 kB 00:00 2023-07-02T08:57:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 111 kB/s | 43 kB 00:00 2023-07-02T08:57:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 584 kB/s | 81 kB 00:00 2023-07-02T08:57:12.815 INFO:teuthology.orchestra.run.smithi164.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 309 kB/s | 155 kB 00:00 2023-07-02T08:57:12.849 INFO:teuthology.orchestra.run.smithi164.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 945 kB/s | 61 kB 00:00 2023-07-02T08:57:13.049 INFO:teuthology.orchestra.run.smithi164.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 262 kB/s | 61 kB 00:00 2023-07-02T08:57:13.166 INFO:teuthology.orchestra.run.smithi164.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.6 MB/s | 1.2 MB 00:00 2023-07-02T08:57:13.169 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:13.169 INFO:teuthology.orchestra.run.smithi164.stdout:Total 639 kB/s | 1.9 MB 00:03 2023-07-02T08:57:13.230 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:57:13.246 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:57:13.246 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:57:13.332 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:57:13.336 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:57:13.909 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:57:14.050 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2023-07-02T08:57:14.190 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-07-02T08:57:14.412 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-07-02T08:57:14.664 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-07-02T08:57:14.887 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:08:53 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:14.910 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-07-02T08:57:15.143 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-07-02T08:57:15.425 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-07-02T08:57:15.812 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-07-02T08:57:16.054 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:57:16.076 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-07-02T08:57:16.206 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:57:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:57:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:57:16.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2023-07-02T08:57:17.161 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-07-02T08:57:17.493 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-07-02T08:57:17.541 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:08:56 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:17.727 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-07-02T08:57:17.971 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-07-02T08:57:18.186 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:18.0.0-4632.g6450d270.el8.noarch 13/13 2023-07-02T08:57:18.480 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4632.g6450d270.el8.noarch 13/13 2023-07-02T08:57:18.480 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4632.g6450d270.el8.noarch 1/13 2023-07-02T08:57:18.480 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-07-02T08:57:18.481 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-07-02T08:57:18.481 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-07-02T08:57:18.481 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-07-02T08:57:18.481 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-07-02T08:57:18.481 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-07-02T08:57:18.481 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-07-02T08:57:18.481 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-07-02T08:57:18.482 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-07-02T08:57:18.482 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-07-02T08:57:18.482 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-07-02T08:57:18.930 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:57:18.931 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:18.931 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:57:18.931 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:18.931 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:57:18.931 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 164 k 2023-07-02T08:57:18.932 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:18.932 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:57:18.932 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:18.932 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2023-07-02T08:57:18.932 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:18.933 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 164 k 2023-07-02T08:57:18.933 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 654 k 2023-07-02T08:57:18.933 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:57:19.201 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-07-02T08:57:19.201 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:19.201 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:57:19.202 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:19.202 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-02T08:57:19.202 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-02T08:57:19.202 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-02T08:57:19.202 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-02T08:57:19.202 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-02T08:57:19.203 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-02T08:57:19.203 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-02T08:57:19.203 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-02T08:57:19.203 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-02T08:57:19.203 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-02T08:57:19.203 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-02T08:57:19.203 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-02T08:57:19.204 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:19.204 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:19.304 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-cephadm 2023-07-02T08:57:19.717 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-18.0.0-4632.g6450d270.el8.noarch.rpm 209 kB/s | 164 kB 00:00 2023-07-02T08:57:19.718 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:19.719 INFO:teuthology.orchestra.run.smithi103.stdout:Total 208 kB/s | 164 kB 00:00 2023-07-02T08:57:19.719 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:57:19.728 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:57:19.728 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:57:19.731 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:57:19.734 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:57:20.129 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:08:59 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:20.310 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:57:21.455 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:21.456 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:21.456 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:57:21.456 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:21.457 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:57:21.457 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 129 k 2023-07-02T08:57:21.457 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:57:21.457 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-02T08:57:21.457 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-07-02T08:57:21.457 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-07-02T08:57:21.457 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-02T08:57:21.458 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:21.458 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:57:21.458 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:21.458 INFO:teuthology.orchestra.run.smithi164.stdout:Install 5 Packages 2023-07-02T08:57:21.458 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:21.459 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 6.0 M 2023-07-02T08:57:21.459 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 26 M 2023-07-02T08:57:21.459 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:57:22.258 INFO:teuthology.orchestra.run.smithi164.stdout:(1/5): ceph-mgr-cephadm-18.0.0-4632.g6450d270.e 232 kB/s | 129 kB 00:00 2023-07-02T08:57:22.291 INFO:teuthology.orchestra.run.smithi164.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 914 kB/s | 538 kB 00:00 2023-07-02T08:57:22.567 INFO:teuthology.orchestra.run.smithi164.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.5 MB/s | 4.8 MB 00:00 2023-07-02T08:57:22.744 INFO:teuthology.orchestra.run.smithi164.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 133 kB/s | 60 kB 00:00 2023-07-02T08:57:22.950 INFO:teuthology.orchestra.run.smithi164.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 687 kB/s | 476 kB 00:00 2023-07-02T08:57:22.952 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:22.953 INFO:teuthology.orchestra.run.smithi164.stdout:Total 4.0 MB/s | 6.0 MB 00:01 2023-07-02T08:57:23.008 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:57:23.019 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:57:23.020 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:57:23.125 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:57:23.128 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:57:23.423 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:57:23.807 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-07-02T08:57:24.823 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-07-02T08:57:25.247 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-07-02T08:57:25.541 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-07-02T08:57:25.785 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4632.g6450d270.el8.noarch 5/5 2023-07-02T08:57:26.070 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4632.g6450d270.el8.noarch 5/5 2023-07-02T08:57:26.070 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/5 2023-07-02T08:57:26.071 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-07-02T08:57:26.071 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-07-02T08:57:26.071 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-07-02T08:57:26.766 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-07-02T08:57:26.767 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:26.767 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:57:26.767 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:26.767 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-02T08:57:26.767 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-02T08:57:26.768 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-02T08:57:26.768 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-02T08:57:26.768 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:26.768 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:26.866 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:57:26.875 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-fuse 2023-07-02T08:57:27.709 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:06 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:28.576 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:57:29.032 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:29.033 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:29.033 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:57:29.034 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:29.034 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:57:29.034 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:18.0.0-4632.g6450d270.el8 ceph 818 k 2023-07-02T08:57:29.034 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-07-02T08:57:29.034 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-07-02T08:57:29.035 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:29.035 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:57:29.035 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:29.035 INFO:teuthology.orchestra.run.smithi164.stdout:Install 2 Packages 2023-07-02T08:57:29.035 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:29.036 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 901 k 2023-07-02T08:57:29.036 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 3.3 M 2023-07-02T08:57:29.036 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:57:29.384 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:57:29.384 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:29.384 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:57:29.385 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:29.385 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:29.385 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:57:29.467 INFO:teuthology.orchestra.run.smithi164.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 213 kB/s | 83 kB 00:00 2023-07-02T08:57:29.593 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2023-07-02T08:57:29.992 INFO:teuthology.orchestra.run.smithi164.stdout:(2/2): ceph-fuse-18.0.0-4632.g6450d270.el8.x86_ 895 kB/s | 818 kB 00:00 2023-07-02T08:57:29.993 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:29.993 INFO:teuthology.orchestra.run.smithi164.stdout:Total 940 kB/s | 901 kB 00:00 2023-07-02T08:57:30.004 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:57:30.026 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:57:30.026 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:57:30.114 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:57:30.118 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:57:30.289 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:57:30.430 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:09:09 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:30.576 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-07-02T08:57:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 2/2 2023-07-02T08:57:31.202 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 2/2 2023-07-02T08:57:31.202 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 1/2 2023-07-02T08:57:31.766 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:57:31.766 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:31.767 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2023-07-02T08:57:31.767 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:31.767 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:57:31.767 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4632.g6450d270.el8 ceph 157 k 2023-07-02T08:57:31.767 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:31.767 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:57:31.767 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:31.768 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2023-07-02T08:57:31.768 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:31.768 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 157 k 2023-07-02T08:57:31.769 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 540 k 2023-07-02T08:57:31.769 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:57:32.205 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-07-02T08:57:32.205 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:32.206 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:57:32.206 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-07-02T08:57:32.206 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:32.206 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:32.399 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-18.0.0-4632.g6450d2 248 kB/s | 157 kB 00:00 2023-07-02T08:57:32.400 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:32.400 INFO:teuthology.orchestra.run.smithi103.stdout:Total 248 kB/s | 157 kB 00:00 2023-07-02T08:57:32.400 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:57:32.401 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-volume 2023-07-02T08:57:32.410 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:57:32.411 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:57:32.459 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:57:32.462 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:57:32.578 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:57:32.768 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4632.g6450d270. 1/1 2023-07-02T08:57:33.181 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4632.g6450d270. 1/1 2023-07-02T08:57:33.237 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:12 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:33.736 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4632.g6450d270. 1/1 2023-07-02T08:57:33.737 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:33.737 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:57:33.737 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:57:33.737 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:33.737 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:57:33.999 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2023-07-02T08:57:34.559 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:34.559 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:34.559 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-07-02T08:57:34.559 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:34.560 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:57:34.560 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 258 k 2023-07-02T08:57:34.560 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:34.560 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:57:34.560 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:34.560 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2023-07-02T08:57:34.560 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:34.561 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 258 k 2023-07-02T08:57:34.561 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 1.2 M 2023-07-02T08:57:34.561 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:57:34.830 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:09:13 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:35.581 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-volume-18.0.0-4632.g6450d270.el8.noarch.rp 253 kB/s | 258 kB 00:01 2023-07-02T08:57:35.581 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:35.582 INFO:teuthology.orchestra.run.smithi164.stdout:Total 253 kB/s | 258 kB 00:01 2023-07-02T08:57:35.582 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:57:35.593 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:57:35.593 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:57:35.633 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:57:35.636 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:57:35.819 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:57:35.997 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:57:36.084 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:57:36.148 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:57:36.149 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:57:36.149 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:57:36.244 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2023-07-02T08:57:36.505 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:57:37.063 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:09:16 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:57:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:37.118 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:57:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:37.118 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:37.609 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados-devel 2023-07-02T08:57:38.412 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:57:38.413 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:38.413 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:57:38.414 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:38.414 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:57:38.414 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 1.7 M 2023-07-02T08:57:38.414 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:57:38.414 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 23 k 2023-07-02T08:57:38.414 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 15 k 2023-07-02T08:57:38.415 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-07-02T08:57:38.415 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-02T08:57:38.415 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-02T08:57:38.415 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:38.415 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:57:38.415 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:57:38.416 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2023-07-02T08:57:38.416 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:38.416 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.0 M 2023-07-02T08:57:38.417 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 8.2 M 2023-07-02T08:57:38.417 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:57:38.457 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:17 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:38.703 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): ceph-prometheus-alerts-18.0.0-4632.g6450 79 kB/s | 15 kB 00:00 2023-07-02T08:57:38.762 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): ceph-grafana-dashboards-18.0.0-4632.g645 95 kB/s | 23 kB 00:00 2023-07-02T08:57:38.779 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 564 kB/s | 43 kB 00:00 2023-07-02T08:57:38.854 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 365 kB/s | 34 kB 00:00 2023-07-02T08:57:38.896 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2023-07-02T08:57:39.422 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): ceph-mgr-dashboard-18.0.0-4632.g6450d270 1.9 MB/s | 1.7 MB 00:00 2023-07-02T08:57:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:39.424 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2023-07-02T08:57:39.437 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:57:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:57:39.448 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:57:39.526 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:57:39.529 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:57:39.772 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:39.772 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:39.773 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2023-07-02T08:57:39.773 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:39.773 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:57:39.773 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:18.0.0-4632.g6450d270.el8 ceph 139 k 2023-07-02T08:57:39.773 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:39.773 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:57:39.774 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:39.774 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2023-07-02T08:57:39.774 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:39.774 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 139 k 2023-07-02T08:57:39.775 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 512 k 2023-07-02T08:57:39.775 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:57:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:57:40.006 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-07-02T08:57:40.158 INFO:teuthology.orchestra.run.smithi164.stdout:librados-devel-18.0.0-4632.g6450d270.el8.x86_64 361 kB/s | 139 kB 00:00 2023-07-02T08:57:40.159 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:40.159 INFO:teuthology.orchestra.run.smithi164.stdout:Total 360 kB/s | 139 kB 00:00 2023-07-02T08:57:40.159 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:57:40.164 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:57:40.164 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:57:40.214 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:57:40.218 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:57:40.218 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-07-02T08:57:40.331 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:57:40.461 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-07-02T08:57:40.640 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4632.g6450d270.el8.n 4/6 2023-07-02T08:57:40.640 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:57:40.933 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:57:40.964 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4632.g6450d270.el8. 5/6 2023-07-02T08:57:41.169 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4632.g6450d270.el8.noarc 6/6 2023-07-02T08:57:41.451 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4632.g6450d270.el8.noarc 6/6 2023-07-02T08:57:41.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4632.g6450d270.el8. 1/6 2023-07-02T08:57:41.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4632.g6450d270.el8.noarc 2/6 2023-07-02T08:57:41.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4632.g6450d270.el8.n 3/6 2023-07-02T08:57:41.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-07-02T08:57:41.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-07-02T08:57:41.487 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:57:41.488 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:41.488 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:57:41.488 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:57:41.488 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:41.488 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:41.723 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs2 2023-07-02T08:57:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-07-02T08:57:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:42.030 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:57:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:42.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-02T08:57:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-02T08:57:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-02T08:57:42.031 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:42.031 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:57:42.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2023-07-02T08:57:42.566 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:21 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:42.971 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:09:21 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:43.732 INFO:teuthology.orchestra.run.smithi164.stdout:Package libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:57:43.884 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:43.885 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:57:43.885 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:43.987 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs-devel 2023-07-02T08:57:44.294 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:57:44.296 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================================ 2023-07-02T08:57:44.296 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:57:44.296 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================================ 2023-07-02T08:57:44.296 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:57:44.296 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 7.4 M 2023-07-02T08:57:44.297 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:57:44.297 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-07-02T08:57:44.297 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-07-02T08:57:44.297 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-07-02T08:57:44.297 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-07-02T08:57:44.297 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-07-02T08:57:44.297 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-07-02T08:57:44.298 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-07-02T08:57:44.298 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:44.298 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:57:44.298 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================================ 2023-07-02T08:57:44.298 INFO:teuthology.orchestra.run.smithi103.stdout:Install 8 Packages 2023-07-02T08:57:44.298 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:44.299 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 35 M 2023-07-02T08:57:44.299 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 210 M 2023-07-02T08:57:44.299 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:57:44.818 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:23 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:45.486 INFO:teuthology.orchestra.run.smithi103.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 5.8 MB/s | 4.7 MB 00:00 2023-07-02T08:57:45.561 INFO:teuthology.orchestra.run.smithi103.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.5 MB/s | 4.9 MB 00:00 2023-07-02T08:57:45.670 INFO:teuthology.orchestra.run.smithi103.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2023-07-02T08:57:45.737 INFO:teuthology.orchestra.run.smithi103.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 15 MB/s | 3.7 MB 00:00 2023-07-02T08:57:45.912 INFO:teuthology.orchestra.run.smithi103.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 3.6 MB/s | 645 kB 00:00 2023-07-02T08:57:45.954 INFO:teuthology.orchestra.run.smithi103.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2023-07-02T08:57:46.148 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:46.149 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:46.149 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2023-07-02T08:57:46.149 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:46.149 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:57:46.149 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:18.0.0-4632.g6450d270.el8 ceph 31 k 2023-07-02T08:57:46.149 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:46.149 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:57:46.150 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:57:46.150 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2023-07-02T08:57:46.150 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:46.150 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 31 k 2023-07-02T08:57:46.151 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 139 k 2023-07-02T08:57:46.151 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:57:46.280 INFO:teuthology.orchestra.run.smithi103.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 22 MB/s | 14 MB 00:00 2023-07-02T08:57:46.578 INFO:teuthology.orchestra.run.smithi164.stdout:libcephfs-devel-18.0.0-4632.g6450d270.el8.x86_6 72 kB/s | 31 kB 00:00 2023-07-02T08:57:46.579 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:46.579 INFO:teuthology.orchestra.run.smithi164.stdout:Total 72 kB/s | 31 kB 00:00 2023-07-02T08:57:46.579 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:57:46.582 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:57:46.583 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:57:46.619 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:57:46.622 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:57:46.723 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:57:46.964 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:57:47.236 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:57:47.770 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:57:47.770 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:47.770 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:57:47.771 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:57:47.771 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:57:47.771 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:47.880 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados2 2023-07-02T08:57:48.696 INFO:teuthology.orchestra.run.smithi103.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-463 1.8 MB/s | 7.4 MB 00:04 2023-07-02T08:57:48.697 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:57:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:Total 8.0 MB/s | 35 MB 00:04 2023-07-02T08:57:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:27 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:48.933 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:57:48.951 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:57:48.951 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:57:49.332 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:57:49.335 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:57:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:57:49.888 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:57:50.024 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-07-02T08:57:50.041 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:50.043 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:57:50.043 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:50.142 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librbd1 2023-07-02T08:57:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-07-02T08:57:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-07-02T08:57:50.972 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:29 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-07-02T08:57:51.407 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-07-02T08:57:52.063 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-07-02T08:57:52.142 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:57:52.282 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-07-02T08:57:52.295 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:52.296 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:57:52.296 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:52.387 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rados 2023-07-02T08:57:52.985 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-07-02T08:57:53.221 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:32 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:53.310 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-07-02T08:57:53.590 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-02T08:57:54.381 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rados-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:57:54.532 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:54.534 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:57:54.534 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:54.635 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rgw 2023-07-02T08:57:55.374 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-02T08:57:55.474 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:34 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:56.633 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-07-02T08:57:56.644 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rgw-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:57:56.797 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:56.798 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:57:56.799 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:56.861 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4632.g6450d27 8/8 2023-07-02T08:57:56.898 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-cephfs 2023-07-02T08:57:57.166 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4632.g6450d27 8/8 2023-07-02T08:57:57.166 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4632.g6450d27 1/8 2023-07-02T08:57:57.167 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-07-02T08:57:57.167 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-07-02T08:57:57.167 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-07-02T08:57:57.167 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-07-02T08:57:57.167 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-07-02T08:57:57.167 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-07-02T08:57:57.717 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:36 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-07-02T08:57:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:57.754 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:57:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:57:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-07-02T08:57:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-07-02T08:57:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.15-4.el8.x86_64 2023-07-02T08:57:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-07-02T08:57:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-02T08:57:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-02T08:57:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-07-02T08:57:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:57:57.756 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:57:57.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2023-07-02T08:57:58.689 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:09:37 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:57:58.885 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-cephfs-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:57:59.039 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:57:59.040 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:57:59.041 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:57:59.140 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rbd 2023-07-02T08:57:59.968 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:38 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:00.008 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:00.009 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:00.010 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:58:00.010 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:00.010 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:00.010 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 47 k 2023-07-02T08:58:00.010 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:58:00.010 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-02T08:58:00.011 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-02T08:58:00.011 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-02T08:58:00.011 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-07-02T08:58:00.011 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-07-02T08:58:00.011 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-02T08:58:00.011 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-07-02T08:58:00.011 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-07-02T08:58:00.011 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-07-02T08:58:00.012 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-07-02T08:58:00.012 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-02T08:58:00.012 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-02T08:58:00.012 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:00.012 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:00.012 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:00.012 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2023-07-02T08:58:00.013 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:00.013 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 1.9 M 2023-07-02T08:58:00.013 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 25 M 2023-07-02T08:58:00.013 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:00.655 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): ceph-mgr-rook-18.0.0-4632.g6450d270.el8 266 kB/s | 47 kB 00:00 2023-07-02T08:58:00.714 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 112 kB/s | 27 kB 00:00 2023-07-02T08:58:00.731 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 79 kB/s | 20 kB 00:00 2023-07-02T08:58:00.975 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 422 kB/s | 110 kB 00:00 2023-07-02T08:58:01.048 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 322 kB/s | 126 kB 00:00 2023-07-02T08:58:01.132 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 396 kB/s | 33 kB 00:00 2023-07-02T08:58:01.163 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rbd-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:58:01.165 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 488 kB/s | 16 kB 00:00 2023-07-02T08:58:01.199 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 331 kB/s | 155 kB 00:00 2023-07-02T08:58:01.241 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.1 MB/s | 81 kB 00:00 2023-07-02T08:58:01.292 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2023-07-02T08:58:01.314 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:58:01.315 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2023-07-02T08:58:01.315 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:58:01.342 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-07-02T08:58:01.367 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 109 kB/s | 43 kB 00:00 2023-07-02T08:58:01.407 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-fuse 2023-07-02T08:58:01.542 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.4 MB/s | 1.2 MB 00:00 2023-07-02T08:58:01.543 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:01.543 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-07-02T08:58:01.603 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:01.619 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:01.619 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:01.705 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:01.708 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:02.210 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:02.233 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:41 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:02.492 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-07-02T08:58:02.681 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-07-02T08:58:02.908 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-07-02T08:58:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-07-02T08:58:03.411 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-07-02T08:58:03.554 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:58:03.554 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2023-07-02T08:58:03.555 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:03.555 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:58:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:18.0.0-4632.g6450d270.el8 ceph 91 k 2023-07-02T08:58:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:03.555 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:58:03.555 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:03.555 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2023-07-02T08:58:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:03.556 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 91 k 2023-07-02T08:58:03.556 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 303 k 2023-07-02T08:58:03.557 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:58:03.644 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-07-02T08:58:03.851 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-fuse-18.0.0-4632.g6450d270.el8.x86_64.rpm 308 kB/s | 91 kB 00:00 2023-07-02T08:58:03.852 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:03.852 INFO:teuthology.orchestra.run.smithi164.stdout:Total 306 kB/s | 91 kB 00:00 2023-07-02T08:58:03.853 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:58:03.862 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:58:03.862 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:58:03.911 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:58:03.914 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:58:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-07-02T08:58:04.045 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:58:04.177 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-07-02T08:58:04.211 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:04.546 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:05.128 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:05.128 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:05.129 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:58:05.129 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:58:05.129 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:05.129 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:58:05.198 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-07-02T08:58:05.371 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-mirror 2023-07-02T08:58:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-07-02T08:58:05.712 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-07-02T08:58:05.906 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-07-02T08:58:06.055 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:18.0.0-4632.g6450d270.el8.noarch 13/13 2023-07-02T08:58:06.210 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:45 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:06.352 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4632.g6450d270.el8.noarch 13/13 2023-07-02T08:58:06.352 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4632.g6450d270.el8.noarch 1/13 2023-07-02T08:58:06.353 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-07-02T08:58:06.353 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-07-02T08:58:06.353 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-07-02T08:58:06.353 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-07-02T08:58:06.353 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-07-02T08:58:06.353 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-07-02T08:58:06.354 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-07-02T08:58:06.354 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-07-02T08:58:06.354 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-07-02T08:58:06.354 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-07-02T08:58:06.354 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-07-02T08:58:07.091 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-07-02T08:58:07.091 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:07.091 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:07.092 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:58:07.092 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-02T08:58:07.092 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-02T08:58:07.092 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-02T08:58:07.092 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-02T08:58:07.092 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-02T08:58:07.093 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-02T08:58:07.093 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-02T08:58:07.093 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-02T08:58:07.093 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-02T08:58:07.093 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-02T08:58:07.093 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-02T08:58:07.094 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-02T08:58:07.094 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:07.094 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:07.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2023-07-02T08:58:07.540 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:58:07.541 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:07.541 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2023-07-02T08:58:07.541 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:07.541 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:58:07.541 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:18.0.0-4632.g6450d270.el8 ceph 3.3 M 2023-07-02T08:58:07.541 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:07.541 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:58:07.542 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:07.542 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2023-07-02T08:58:07.542 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:07.542 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 3.3 M 2023-07-02T08:58:07.543 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 18 M 2023-07-02T08:58:07.543 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:58:08.024 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:09:47 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:08.205 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-mirror-18.0.0-4632.g6450d270.el8.x86_64.rpm 5.0 MB/s | 3.3 MB 00:00 2023-07-02T08:58:08.205 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:08.206 INFO:teuthology.orchestra.run.smithi164.stdout:Total 5.0 MB/s | 3.3 MB 00:00 2023-07-02T08:58:08.206 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:58:08.218 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:58:08.218 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:58:08.281 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:58:08.284 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:58:08.728 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:58:08.945 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:09.337 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:09.338 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:09.338 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:58:09.338 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:09.338 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:09.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 129 k 2023-07-02T08:58:09.338 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:58:09.339 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-02T08:58:09.339 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-07-02T08:58:09.339 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-07-02T08:58:09.339 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-02T08:58:09.339 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:09.339 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:09.339 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:09.340 INFO:teuthology.orchestra.run.smithi103.stdout:Install 5 Packages 2023-07-02T08:58:09.340 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:09.340 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 6.0 M 2023-07-02T08:58:09.340 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2023-07-02T08:58:09.341 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:09.348 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:09.843 INFO:teuthology.orchestra.run.smithi103.stdout:(1/5): ceph-mgr-cephadm-18.0.0-4632.g6450d270.e 443 kB/s | 129 kB 00:00 2023-07-02T08:58:09.956 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:09.957 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:09.957 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:58:09.957 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:58:09.957 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:09.957 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:58:10.102 INFO:teuthology.orchestra.run.smithi103.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 979 kB/s | 538 kB 00:00 2023-07-02T08:58:10.127 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-nbd 2023-07-02T08:58:10.128 INFO:teuthology.orchestra.run.smithi103.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.6 MB/s | 476 kB 00:00 2023-07-02T08:58:10.386 INFO:teuthology.orchestra.run.smithi103.stdout:(4/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.7 MB/s | 4.8 MB 00:00 2023-07-02T08:58:10.411 INFO:teuthology.orchestra.run.smithi103.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 195 kB/s | 60 kB 00:00 2023-07-02T08:58:10.411 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:10.411 INFO:teuthology.orchestra.run.smithi103.stdout:Total 5.6 MB/s | 6.0 MB 00:01 2023-07-02T08:58:10.463 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:10.474 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:10.475 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:10.578 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:10.581 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:10.866 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:10.970 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:09:49 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:11.211 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-07-02T08:58:12.304 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-07-02T08:58:12.305 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:12.305 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2023-07-02T08:58:12.305 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:12.306 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-07-02T08:58:12.306 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:18.0.0-4632.g6450d270.el8 ceph 177 k 2023-07-02T08:58:12.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:12.306 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-07-02T08:58:12.306 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-07-02T08:58:12.306 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2023-07-02T08:58:12.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:12.307 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 177 k 2023-07-02T08:58:12.308 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 626 k 2023-07-02T08:58:12.308 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-07-02T08:58:12.357 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-07-02T08:58:12.527 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-nbd-18.0.0-4632.g6450d270.el8.x86_64.rpm 804 kB/s | 177 kB 00:00 2023-07-02T08:58:12.527 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:12.528 INFO:teuthology.orchestra.run.smithi164.stdout:Total 799 kB/s | 177 kB 00:00 2023-07-02T08:58:12.528 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-07-02T08:58:12.538 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-07-02T08:58:12.538 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-07-02T08:58:12.588 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-07-02T08:58:12.591 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-07-02T08:58:12.705 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-07-02T08:58:12.718 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-07-02T08:58:12.926 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-07-02T08:58:13.204 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4632.g6450d270.el8.noarch 5/5 2023-07-02T08:58:13.215 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-nbd-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4632.g6450d270.el8.noarch 5/5 2023-07-02T08:58:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4632.g6450d270.el8.noarch 1/5 2023-07-02T08:58:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-07-02T08:58:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-07-02T08:58:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-07-02T08:58:13.812 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:13.812 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:13.812 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-07-02T08:58:13.812 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:58:13.812 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:58:13.813 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-07-02T08:58:13.990 DEBUG:teuthology.parallel:result is None 2023-07-02T08:58:14.054 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-07-02T08:58:14.054 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:14.054 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:14.054 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:58:14.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-02T08:58:14.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-02T08:58:14.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-02T08:58:14.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-02T08:58:14.055 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:14.055 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:14.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2023-07-02T08:58:15.009 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:09:54 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:16.329 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:16.330 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:58:16.330 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:16.330 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:18.0.0-4632.g6450d270.el8 ceph 818 k 2023-07-02T08:58:16.331 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2023-07-02T08:58:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-07-02T08:58:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:16.331 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:16.331 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:16.331 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2023-07-02T08:58:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:16.332 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 901 k 2023-07-02T08:58:16.332 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 3.3 M 2023-07-02T08:58:16.332 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:16.895 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): ceph-fuse-18.0.0-4632.g6450d270.el8.x86_ 2.1 MB/s | 818 kB 00:00 2023-07-02T08:58:16.920 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 207 kB/s | 83 kB 00:00 2023-07-02T08:58:16.921 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:16.921 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.5 MB/s | 901 kB 00:00 2023-07-02T08:58:16.931 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:16.952 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:16.952 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:17.038 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:17.041 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:17.206 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:17.494 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-07-02T08:58:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 2/2 2023-07-02T08:58:18.097 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 2/2 2023-07-02T08:58:18.097 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 1/2 2023-07-02T08:58:18.684 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-07-02T08:58:18.685 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:18.685 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:18.685 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-07-02T08:58:18.685 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:18.685 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:18.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-volume 2023-07-02T08:58:19.696 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:09:58 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:21.017 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:21.018 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:21.018 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2023-07-02T08:58:21.018 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:21.018 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:21.018 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:18.0.0-4632.g6450d270.el8 ceph-noarch 258 k 2023-07-02T08:58:21.018 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:21.019 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:21.019 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:21.019 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2023-07-02T08:58:21.019 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:21.019 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 258 k 2023-07-02T08:58:21.019 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.2 M 2023-07-02T08:58:21.020 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:21.226 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-volume-18.0.0-4632.g6450d270.el8.noarch.rp 1.2 MB/s | 258 kB 00:00 2023-07-02T08:58:21.227 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:21.227 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.2 MB/s | 258 kB 00:00 2023-07-02T08:58:21.227 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:21.237 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:21.237 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:21.272 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:21.275 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:22.712 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:58:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:58:23.692 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:18.0.0-4632.g6450d270.el8.noarch 1/1 2023-07-02T08:58:23.692 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:23.692 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:23.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:18.0.0-4632.g6450d270.el8.noarch 2023-07-02T08:58:23.693 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:23.693 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:23.908 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2023-07-02T08:58:24.754 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:03 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:26.078 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:26.079 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:26.079 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2023-07-02T08:58:26.079 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:26.080 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:26.080 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:18.0.0-4632.g6450d270.el8 ceph 139 k 2023-07-02T08:58:26.080 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:26.080 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:26.080 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:26.080 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2023-07-02T08:58:26.080 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:26.081 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 139 k 2023-07-02T08:58:26.081 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 512 k 2023-07-02T08:58:26.081 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:26.266 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-18.0.0-4632.g6450d270.el8.x86_64 748 kB/s | 139 kB 00:00 2023-07-02T08:58:26.266 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:26.267 INFO:teuthology.orchestra.run.smithi103.stdout:Total 743 kB/s | 139 kB 00:00 2023-07-02T08:58:26.267 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:26.272 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:26.272 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:26.321 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:26.325 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:26.992 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:27.555 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:27.556 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:27.556 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:27.556 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:58:27.556 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:27.556 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:28.608 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2023-07-02T08:58:29.450 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:08 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:30.621 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:58:30.774 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:30.775 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:58:30.775 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:30.884 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2023-07-02T08:58:31.720 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:10 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:33.028 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:33.029 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:33.029 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2023-07-02T08:58:33.029 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:33.030 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:33.030 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:18.0.0-4632.g6450d270.el8 ceph 31 k 2023-07-02T08:58:33.030 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:33.030 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:33.030 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:33.030 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2023-07-02T08:58:33.031 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:33.031 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 31 k 2023-07-02T08:58:33.031 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 139 k 2023-07-02T08:58:33.031 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:33.266 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-18.0.0-4632.g6450d270.el8.x86_6 131 kB/s | 31 kB 00:00 2023-07-02T08:58:33.267 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:33.267 INFO:teuthology.orchestra.run.smithi103.stdout:Total 131 kB/s | 31 kB 00:00 2023-07-02T08:58:33.267 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:33.270 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:33.271 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:33.300 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:33.303 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:33.357 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:33.844 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:34.376 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:34.376 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:34.377 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:34.377 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:58:34.377 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:34.377 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:34.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2023-07-02T08:58:35.319 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:14 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:36.502 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:58:36.654 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:36.655 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:58:36.655 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:36.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2023-07-02T08:58:37.579 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:16 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:38.745 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:58:38.898 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:38.899 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:58:38.899 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:39.017 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2023-07-02T08:58:39.847 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:18 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:41.019 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:58:41.172 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:41.173 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:58:41.173 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:41.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2023-07-02T08:58:42.098 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:21 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:58:43.432 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:43.433 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:58:43.433 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:43.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2023-07-02T08:58:44.352 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:23 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:45.527 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:58:45.677 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:45.679 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:58:45.679 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:45.776 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2023-07-02T08:58:46.605 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:25 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:47.761 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:18.0.0-4632.g6450d270.el8.x86_64 is already installed. 2023-07-02T08:58:47.914 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:47.915 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2023-07-02T08:58:47.915 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:48.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2023-07-02T08:58:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:27 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:50.162 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:50.162 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2023-07-02T08:58:50.163 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:50.163 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:18.0.0-4632.g6450d270.el8 ceph 91 k 2023-07-02T08:58:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:50.163 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:50.164 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:50.164 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2023-07-02T08:58:50.164 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:50.164 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 91 k 2023-07-02T08:58:50.165 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 303 k 2023-07-02T08:58:50.165 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:50.332 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-18.0.0-4632.g6450d270.el8.x86_64.rpm 540 kB/s | 91 kB 00:00 2023-07-02T08:58:50.333 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:50.333 INFO:teuthology.orchestra.run.smithi103.stdout:Total 536 kB/s | 91 kB 00:00 2023-07-02T08:58:50.334 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:50.343 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:50.343 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:50.391 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:50.394 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:50.731 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:51.024 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:51.590 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:51.590 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:51.590 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:51.590 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:58:51.591 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:51.591 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:51.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2023-07-02T08:58:52.634 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:31 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:53.971 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:53.972 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:53.972 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2023-07-02T08:58:53.972 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:53.972 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:53.972 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:18.0.0-4632.g6450d270.el8 ceph 3.3 M 2023-07-02T08:58:53.972 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:53.973 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:53.973 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:53.973 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2023-07-02T08:58:53.973 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:53.974 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 3.3 M 2023-07-02T08:58:53.974 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 18 M 2023-07-02T08:58:53.974 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:54.345 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-18.0.0-4632.g6450d270.el8.x86_64.rpm 8.9 MB/s | 3.3 MB 00:00 2023-07-02T08:58:54.345 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:54.346 INFO:teuthology.orchestra.run.smithi103.stdout:Total 8.9 MB/s | 3.3 MB 00:00 2023-07-02T08:58:54.346 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:54.357 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:54.357 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:54.417 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:54.421 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:54.869 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:55.049 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:55.473 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:56.053 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:56.054 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:56.054 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:56.054 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:58:56.054 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:56.054 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:58:56.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2023-07-02T08:58:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:36 ago on Sun 02 Jul 2023 08:48:21 AM UTC. 2023-07-02T08:58:58.455 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2023-07-02T08:58:58.456 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:58.456 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2023-07-02T08:58:58.456 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:58.456 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2023-07-02T08:58:58.456 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:18.0.0-4632.g6450d270.el8 ceph 177 k 2023-07-02T08:58:58.456 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:58.456 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2023-07-02T08:58:58.457 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2023-07-02T08:58:58.457 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2023-07-02T08:58:58.457 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:58.457 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 177 k 2023-07-02T08:58:58.458 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 626 k 2023-07-02T08:58:58.458 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2023-07-02T08:58:58.633 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-18.0.0-4632.g6450d270.el8.x86_64.rpm 1.0 MB/s | 177 kB 00:00 2023-07-02T08:58:58.634 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2023-07-02T08:58:58.634 INFO:teuthology.orchestra.run.smithi103.stdout:Total 997 kB/s | 177 kB 00:00 2023-07-02T08:58:58.635 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2023-07-02T08:58:58.645 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2023-07-02T08:58:58.645 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2023-07-02T08:58:58.696 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2023-07-02T08:58:58.699 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2023-07-02T08:58:58.818 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2023-07-02T08:58:59.022 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:59.323 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:59.896 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:18.0.0-4632.g6450d270.el8.x86_64 1/1 2023-07-02T08:58:59.896 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:59.896 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2023-07-02T08:58:59.896 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:18.0.0-4632.g6450d270.el8.x86_64 2023-07-02T08:58:59.897 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:58:59.897 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2023-07-02T08:59:00.101 DEBUG:teuthology.parallel:result is None 2023-07-02T08:59:00.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=6450d2709f998063da1a187d826fea699691b0b1 2023-07-02T08:59:00.342 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-02T08:59:00.364 INFO:teuthology.orchestra.run.smithi103.stdout:18.0.0-4632.g6450d270.el8 2023-07-02T08:59:00.365 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4632.g6450d270.el8 2023-07-02T08:59:00.365 INFO:teuthology.task.install:The correct ceph version 18.0.0-4632.g6450d270 is installed. 2023-07-02T08:59:00.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=6450d2709f998063da1a187d826fea699691b0b1 2023-07-02T08:59:00.507 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-02T08:59:00.535 INFO:teuthology.orchestra.run.smithi164.stdout:18.0.0-4632.g6450d270.el8 2023-07-02T08:59:00.536 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4632.g6450d270.el8 2023-07-02T08:59:00.536 INFO:teuthology.task.install:The correct ceph version 18.0.0-4632.g6450d270 is installed. 2023-07-02T08:59:00.536 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-02T08:59:00.537 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:00.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-02T08:59:00.567 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:00.567 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-02T08:59:00.608 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-02T08:59:00.608 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:00.608 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2023-07-02T08:59:00.636 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-02T08:59:00.703 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:00.703 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2023-07-02T08:59:00.732 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-02T08:59:00.800 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-02T08:59:00.801 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:00.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-02T08:59:00.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-02T08:59:00.894 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:00.894 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-02T08:59:00.923 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-02T08:59:00.991 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-02T08:59:00.991 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:00.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2023-07-02T08:59:01.017 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-02T08:59:01.084 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:01.085 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2023-07-02T08:59:01.112 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-02T08:59:01.179 INFO:teuthology.run_tasks:Running task ceph... 2023-07-02T08:59:01.253 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-07-02T08:59:01.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2023-07-02T08:59:01.255 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2023-07-02T08:59:01.282 INFO:tasks.ceph:Disabling ceph logrotate... 2023-07-02T08:59:01.283 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-02T08:59:01.322 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-02T08:59:01.348 INFO:tasks.ceph:Creating extra log directories... 2023-07-02T08:59:01.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-02T08:59:01.389 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-02T08:59:01.424 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-07-02T08:59:01.424 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-07-02T08:59:01.425 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293', 'branch': 'wip-yingxin-msgr-multi-core-crimson-10', 'description': 'crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7324293', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-msgr-multi-core-crimson-10'}, 'ceph': {'conf': {'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '6450d2709f998063da1a187d826fea699691b0b1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '6450d2709f998063da1a187d826fea699691b0b1'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '00edceddba022d4d35c4fe2bf8ae4808b752654c'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '6450d2709f998063da1a187d826fea699691b0b1', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '00edceddba022d4d35c4fe2bf8ae4808b752654c', 'targets': {'smithi103.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOqS6e1wxD0IKfQGrjPbQM7M7V4/e4hjGoWI/J5FO15c51f7uUO96HBA/cGQVUVNUmlre9YDyklL2gyLumPXPMBNP7LF4W9ordzlTcNmfKk7eMOQzbpGdVrDmPmb4jSA72Bm2LTYvspP0U1c7eFXvjM4mRYjHjl5IsdCTbdTSFhtR+pYPw6pk/zOX6IuISt9HALOtvguv3x/5x3q1+C7mUFIZMeJUjEoBrY2VHmoPtiWdeqs6PplgEwPBKUqG0PaYhnvMMN2yFrieUj4oAfEiUQ/VXMB1sS9cBCWCA2DgX4FhcF4NMuE96EPPjorl6hx6J43GWGEQHKyMTP/ptmuuGVDOInrAvlgJE5BpNKrtarM1TDFfCJy/MqwTVT/1L8n3wBIug6G1yis0lgmYIoZrrR4xmkThpiqB1WGkidBgeh/Ye2Gwv8JjELSuk+c4UH1PUW49yVeqQ4bCj/aKj6l+3V/Ty9vQa5QivjsgEqUs1SWtMfOcgBGpyH6nxmutQVoE=', 'smithi164.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj/UFFLKldYHPXfTfKh85+du+x3iJh4dlxPOVHYlmglEVLCsVpbRknPbArvu1MyXmisW5Lknb9KYL7Xi6h+EvZfYiLpDdD4CoJAkrb73ldcQ9OnH5tPac+mDM/D4AxS5zv2uiRXWiBqgCS2vwiUpdh0eZFnCtysnBpzAYU5tyjdjKdGWjCvD2i+5DfJs/ciqeDW3coTuRx2GjwSYEwGlQ3w3toVgzayP25ZnzYIrK9sOQD4ZbxBnoWbt9//GEEcQ9G7DEP3g7tfuQAjy655jwS0R6eedNDDTFqrBVqVvKX2hZvGl8icnnSmeUzi7srcIkH7Xmrl98Jn4CR39/iwlcJsEk20+IanaAMwwmnV6leKhaDOExYc2bsk9/ZTHuv5Ufi8gRQbkwLTZoVIrveYsgXCCwf4lScUiKnTqJ9QhVzf9ZbOMXL/HYChzi1PmaYKX7d4bRqkyICyEGChqYqTWyB/DIURFVgkkP69UCBo+YoyyoKpn710e+1I/YG5VOe9xE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '6450d2709f998063da1a187d826fea699691b0b1', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rados/test.sh', 'rados/test_pool_quota.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b53f236c1077f1d295baafb9e0ca9a4b447019d7', 'timestamp': '2023-07-02_08:25:06', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-07-02T08:59:01.425 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-07-02T08:59:01.456 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-07-02T08:59:01.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2023-07-02T08:59:01.514 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2023-07-02T08:59:01.540 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:01.541 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2023-07-02T08:59:01.596 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-02T08:59:01.597 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2023-07-02T08:59:01.655 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-02T08:59:01.655 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T08:59:01.655 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 384053 Links: 1 2023-07-02T08:59:01.655 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T08:59:01.655 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T08:59:01.655 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2023-07-02 08:58:55.225877013 +0000 2023-07-02T08:59:01.655 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2023-07-02 08:47:11.153355797 +0000 2023-07-02T08:59:01.656 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2023-07-02 08:47:11.153355797 +0000 2023-07-02T08:59:01.656 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2023-07-02T08:59:01.656 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-02T08:59:01.720 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2023-07-02T08:59:01.721 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2023-07-02T08:59:01.721 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000248826 s, 2.1 MB/s 2023-07-02T08:59:01.722 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-02T08:59:01.781 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2023-07-02T08:59:01.838 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-02T08:59:01.838 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T08:59:01.839 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 383417 Links: 1 2023-07-02T08:59:01.839 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T08:59:01.839 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T08:59:01.839 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2023-07-02 08:58:55.225877013 +0000 2023-07-02T08:59:01.839 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2023-07-02 08:47:11.672346185 +0000 2023-07-02T08:59:01.839 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2023-07-02 08:47:11.672346185 +0000 2023-07-02T08:59:01.839 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2023-07-02T08:59:01.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-02T08:59:01.903 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2023-07-02T08:59:01.904 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2023-07-02T08:59:01.904 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000219928 s, 2.3 MB/s 2023-07-02T08:59:01.904 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-02T08:59:01.964 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2023-07-02T08:59:02.022 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-02T08:59:02.022 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T08:59:02.022 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 384490 Links: 1 2023-07-02T08:59:02.022 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T08:59:02.022 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T08:59:02.023 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2023-07-02 08:58:55.225877013 +0000 2023-07-02T08:59:02.023 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2023-07-02 08:47:12.195336500 +0000 2023-07-02T08:59:02.023 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2023-07-02 08:47:12.195336500 +0000 2023-07-02T08:59:02.023 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2023-07-02T08:59:02.023 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-02T08:59:02.088 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2023-07-02T08:59:02.089 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2023-07-02T08:59:02.089 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000218033 s, 2.3 MB/s 2023-07-02T08:59:02.089 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-02T08:59:02.148 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2023-07-02T08:59:02.206 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-02T08:59:02.207 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T08:59:02.207 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 383868 Links: 1 2023-07-02T08:59:02.207 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T08:59:02.207 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T08:59:02.207 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2023-07-02 08:58:55.225877013 +0000 2023-07-02T08:59:02.207 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2023-07-02 08:47:12.723326722 +0000 2023-07-02T08:59:02.207 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2023-07-02 08:47:12.723326722 +0000 2023-07-02T08:59:02.208 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2023-07-02T08:59:02.208 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-02T08:59:02.272 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2023-07-02T08:59:02.272 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2023-07-02T08:59:02.272 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000229984 s, 2.2 MB/s 2023-07-02T08:59:02.273 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-02T08:59:02.331 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-07-02T08:59:02.331 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:02.332 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2023-07-02T08:59:02.351 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-02T08:59:02.352 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2023-07-02T08:59:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-02T08:59:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T08:59:02.411 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 377736 Links: 1 2023-07-02T08:59:02.411 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T08:59:02.411 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T08:59:02.411 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-07-02 08:58:09.100452341 +0000 2023-07-02T08:59:02.411 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-07-02 08:47:12.244346361 +0000 2023-07-02T08:59:02.411 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-07-02 08:47:12.244346361 +0000 2023-07-02T08:59:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2023-07-02T08:59:02.412 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-02T08:59:02.478 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-07-02T08:59:02.478 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-07-02T08:59:02.478 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000250268 s, 2.0 MB/s 2023-07-02T08:59:02.479 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-02T08:59:02.539 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2023-07-02T08:59:02.597 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-02T08:59:02.597 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T08:59:02.598 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 374536 Links: 1 2023-07-02T08:59:02.598 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T08:59:02.598 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T08:59:02.598 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-07-02 08:58:09.100452341 +0000 2023-07-02T08:59:02.598 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-07-02 08:47:12.829336938 +0000 2023-07-02T08:59:02.598 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-07-02 08:47:12.829336938 +0000 2023-07-02T08:59:02.598 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2023-07-02T08:59:02.599 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-02T08:59:02.664 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-07-02T08:59:02.664 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-07-02T08:59:02.664 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000219436 s, 2.3 MB/s 2023-07-02T08:59:02.665 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-02T08:59:02.724 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2023-07-02T08:59:02.782 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-02T08:59:02.782 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T08:59:02.782 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 374608 Links: 1 2023-07-02T08:59:02.782 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T08:59:02.783 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T08:59:02.783 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-07-02 08:58:09.100452341 +0000 2023-07-02T08:59:02.783 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-07-02 08:47:13.367328272 +0000 2023-07-02T08:59:02.783 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-07-02 08:47:13.367328272 +0000 2023-07-02T08:59:02.783 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2023-07-02T08:59:02.783 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-02T08:59:02.847 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-07-02T08:59:02.848 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-07-02T08:59:02.848 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000218192 s, 2.3 MB/s 2023-07-02T08:59:02.848 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-02T08:59:02.907 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2023-07-02T08:59:02.964 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-02T08:59:02.964 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T08:59:02.964 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 379443 Links: 1 2023-07-02T08:59:02.965 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T08:59:02.965 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T08:59:02.965 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-07-02 08:58:09.100452341 +0000 2023-07-02T08:59:02.965 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-07-02 08:47:13.893319800 +0000 2023-07-02T08:59:02.965 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-07-02 08:47:13.893319800 +0000 2023-07-02T08:59:02.965 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2023-07-02T08:59:02.966 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-02T08:59:03.030 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-07-02T08:59:03.030 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-07-02T08:59:03.030 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000230202 s, 2.2 MB/s 2023-07-02T08:59:03.031 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-02T08:59:03.089 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-07-02T08:59:03.090 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-07-02T08:59:03.090 INFO:tasks.ceph:Generating config... 2023-07-02T08:59:03.091 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-07-02T08:59:03.091 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-07-02T08:59:03.092 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-07-02T08:59:03.092 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-07-02T08:59:03.092 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-07-02T08:59:03.092 INFO:tasks.ceph:[mon] debug mon = 20 2023-07-02T08:59:03.093 INFO:tasks.ceph:[mon] debug ms = 1 2023-07-02T08:59:03.093 INFO:tasks.ceph:[mon] debug paxos = 20 2023-07-02T08:59:03.093 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-07-02T08:59:03.093 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-07-02T08:59:03.093 INFO:tasks.ceph:[osd] debug monc = 20 2023-07-02T08:59:03.094 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-07-02T08:59:03.094 INFO:tasks.ceph:[osd] debug ms = 1 2023-07-02T08:59:03.094 INFO:tasks.ceph:[osd] debug osd = 20 2023-07-02T08:59:03.094 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-07-02T08:59:03.094 INFO:tasks.ceph:[osd] osd class default list = * 2023-07-02T08:59:03.094 INFO:tasks.ceph:[osd] osd class load list = * 2023-07-02T08:59:03.095 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-07-02T08:59:03.095 INFO:tasks.ceph:[client] debug ms = 1 2023-07-02T08:59:03.095 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-07-02T08:59:03.095 INFO:tasks.ceph:[global] ms client mode = crc 2023-07-02T08:59:03.095 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-07-02T08:59:03.096 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-07-02T08:59:03.096 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-07-02T08:59:03.096 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-07-02T08:59:03.096 INFO:tasks.ceph:[global] ms service mode = crc 2023-07-02T08:59:03.096 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-07-02T08:59:03.097 INFO:tasks.ceph:[mgr] debug ms = 1 2023-07-02T08:59:03.097 INFO:tasks.ceph:Setting up mon.a... 2023-07-02T08:59:03.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-07-02T08:59:03.165 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.keyring 2023-07-02T08:59:03.169 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-07-02T08:59:03.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-02T08:59:03.250 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.103')] 2023-07-02T08:59:03.251 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.103', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug ms': 1}, 'mon.a': {}} 2023-07-02T08:59:03.251 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:03.252 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-07-02T08:59:03.307 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.103 --print /home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:03.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:03.396+0000 7fa29eee1080 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:03.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:03.396+0000 7fa29eee1080 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:03.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:03.396+0000 7fa29eee1080 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:03.398 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:03.398 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: generated fsid 334b59df-ef47-4729-a173-1813560f60f2 2023-07-02T08:59:03.398 INFO:teuthology.orchestra.run.smithi103.stdout:setting min_mon_release = pacific 2023-07-02T08:59:03.398 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 0 2023-07-02T08:59:03.398 INFO:teuthology.orchestra.run.smithi103.stdout:fsid 334b59df-ef47-4729-a173-1813560f60f2 2023-07-02T08:59:03.398 INFO:teuthology.orchestra.run.smithi103.stdout:last_changed 2023-07-02T08:59:03.398196+0000 2023-07-02T08:59:03.399 INFO:teuthology.orchestra.run.smithi103.stdout:created 2023-07-02T08:59:03.398196+0000 2023-07-02T08:59:03.399 INFO:teuthology.orchestra.run.smithi103.stdout:min_mon_release 16 (pacific) 2023-07-02T08:59:03.399 INFO:teuthology.orchestra.run.smithi103.stdout:election_strategy: 1 2023-07-02T08:59:03.399 INFO:teuthology.orchestra.run.smithi103.stdout:0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2023-07-02T08:59:03.399 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-07-02T08:59:03.400 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-07-02T08:59:03.417 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 334b59df-ef47-4729-a173-1813560f60f2... 2023-07-02T08:59:03.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-07-02T08:59:03.459 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-07-02T08:59:03.501 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2023-07-02T08:59:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2023-07-02T08:59:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-02T08:59:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2023-07-02T08:59:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2023-07-02T08:59:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2023-07-02T08:59:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2023-07-02T08:59:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2023-07-02T08:59:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2023-07-02T08:59:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2023-07-02T08:59:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-02T08:59:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2023-07-02T08:59:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2023-07-02T08:59:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-02T08:59:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2023-07-02T08:59:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2023-07-02T08:59:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2023-07-02T08:59:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2023-07-02T08:59:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2023-07-02T08:59:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2023-07-02T08:59:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2023-07-02T08:59:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2023-07-02T08:59:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-02T08:59:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-02T08:59:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2023-07-02T08:59:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-02T08:59:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2023-07-02T08:59:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2023-07-02T08:59:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2023-07-02T08:59:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2023-07-02T08:59:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2023-07-02T08:59:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = 172.21.15.103 2023-07-02T08:59:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-07-02T08:59:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: ms client mode = crc 2023-07-02T08:59:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: ms cluster mode = crc 2023-07-02T08:59:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon client mode = crc 2023-07-02T08:59:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon cluster mode = crc 2023-07-02T08:59:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon service mode = crc 2023-07-02T08:59:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: ms service mode = crc 2023-07-02T08:59:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 334b59df-ef47-4729-a173-1813560f60f2 2023-07-02T08:59:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.509 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2023-07-02T08:59:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2023-07-02T08:59:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2023-07-02T08:59:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2023-07-02T08:59:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2023-07-02T08:59:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2023-07-02T08:59:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2023-07-02T08:59:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2023-07-02T08:59:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2023-07-02T08:59:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2023-07-02T08:59:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2023-07-02T08:59:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2023-07-02T08:59:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2023-07-02T08:59:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2023-07-02T08:59:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2023-07-02T08:59:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2023-07-02T08:59:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2023-07-02T08:59:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2023-07-02T08:59:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2023-07-02T08:59:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: debug monc = 20 2023-07-02T08:59:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar smp = 3 2023-07-02T08:59:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2023-07-02T08:59:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2023-07-02T08:59:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: osd blocked scrub grace period = 3600 2023-07-02T08:59:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: osd class default list = * 2023-07-02T08:59:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: osd class load list = * 2023-07-02T08:59:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: osd shutdown pgref assert = True 2023-07-02T08:59:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.513 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2023-07-02T08:59:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2023-07-02T08:59:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2023-07-02T08:59:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2023-07-02T08:59:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2023-07-02T08:59:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2023-07-02T08:59:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2023-07-02T08:59:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2023-07-02T08:59:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.514 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2023-07-02T08:59:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2023-07-02T08:59:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2023-07-02T08:59:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2023-07-02T08:59:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2023-07-02T08:59:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2023-07-02T08:59:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2023-07-02T08:59:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2023-07-02T08:59:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2023-07-02T08:59:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2023-07-02T08:59:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2023-07-02T08:59:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-02T08:59:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2023-07-02T08:59:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2023-07-02T08:59:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2023-07-02T08:59:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-02T08:59:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-02T08:59:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2023-07-02T08:59:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2023-07-02T08:59:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2023-07-02T08:59:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: mon min osdmap epochs = 50 2023-07-02T08:59:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune interval = 2 2023-07-02T08:59:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune min = 15 2023-07-02T08:59:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune txsize = 2 2023-07-02T08:59:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: paxos service trim min = 10 2023-07-02T08:59:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on pool no app = False 2023-07-02T08:59:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default crimson = True 2023-07-02T08:59:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:03.518 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2023-07-02T08:59:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2023-07-02T08:59:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2023-07-02T08:59:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2023-07-02T08:59:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-02T08:59:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-02T08:59:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2023-07-02T08:59:03.519 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2023-07-02T08:59:03.520 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2023-07-02T08:59:03.521 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2023-07-02T08:59:03.521 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-02T08:59:03.521 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2023-07-02T08:59:03.521 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.521 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2023-07-02T08:59:03.521 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.522 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2023-07-02T08:59:03.522 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.522 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2023-07-02T08:59:03.522 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2023-07-02T08:59:03.522 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.522 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2023-07-02T08:59:03.522 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2023-07-02T08:59:03.522 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.523 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-02T08:59:03.523 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2023-07-02T08:59:03.523 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.523 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2023-07-02T08:59:03.523 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-02T08:59:03.523 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.523 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2023-07-02T08:59:03.523 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.524 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2023-07-02T08:59:03.524 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2023-07-02T08:59:03.524 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2023-07-02T08:59:03.524 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2023-07-02T08:59:03.524 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2023-07-02T08:59:03.524 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2023-07-02T08:59:03.524 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2023-07-02T08:59:03.525 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-02T08:59:03.525 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-02T08:59:03.525 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2023-07-02T08:59:03.525 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.525 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-02T08:59:03.525 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.525 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2023-07-02T08:59:03.525 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.526 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2023-07-02T08:59:03.526 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.526 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2023-07-02T08:59:03.526 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2023-07-02T08:59:03.526 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2023-07-02T08:59:03.526 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = 172.21.15.103 2023-07-02T08:59:03.526 INFO:teuthology.orchestra.run.smithi164.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-07-02T08:59:03.526 INFO:teuthology.orchestra.run.smithi164.stdout: ms client mode = crc 2023-07-02T08:59:03.527 INFO:teuthology.orchestra.run.smithi164.stdout: ms cluster mode = crc 2023-07-02T08:59:03.527 INFO:teuthology.orchestra.run.smithi164.stdout: ms mon client mode = crc 2023-07-02T08:59:03.527 INFO:teuthology.orchestra.run.smithi164.stdout: ms mon cluster mode = crc 2023-07-02T08:59:03.527 INFO:teuthology.orchestra.run.smithi164.stdout: ms mon service mode = crc 2023-07-02T08:59:03.527 INFO:teuthology.orchestra.run.smithi164.stdout: ms service mode = crc 2023-07-02T08:59:03.527 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 334b59df-ef47-4729-a173-1813560f60f2 2023-07-02T08:59:03.527 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.527 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2023-07-02T08:59:03.528 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2023-07-02T08:59:03.528 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.528 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2023-07-02T08:59:03.528 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2023-07-02T08:59:03.528 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2023-07-02T08:59:03.528 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.528 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2023-07-02T08:59:03.528 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2023-07-02T08:59:03.529 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.529 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2023-07-02T08:59:03.529 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2023-07-02T08:59:03.529 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2023-07-02T08:59:03.529 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.529 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2023-07-02T08:59:03.529 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2023-07-02T08:59:03.530 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.530 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2023-07-02T08:59:03.530 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.530 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2023-07-02T08:59:03.530 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.530 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2023-07-02T08:59:03.530 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.530 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2023-07-02T08:59:03.531 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2023-07-02T08:59:03.531 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.531 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2023-07-02T08:59:03.531 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2023-07-02T08:59:03.531 INFO:teuthology.orchestra.run.smithi164.stdout: debug monc = 20 2023-07-02T08:59:03.531 INFO:teuthology.orchestra.run.smithi164.stdout: crimson seastar smp = 3 2023-07-02T08:59:03.531 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-07-02T08:59:03.531 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2023-07-02T08:59:03.532 INFO:teuthology.orchestra.run.smithi164.stdout: osd blocked scrub grace period = 3600 2023-07-02T08:59:03.532 INFO:teuthology.orchestra.run.smithi164.stdout: osd class default list = * 2023-07-02T08:59:03.532 INFO:teuthology.orchestra.run.smithi164.stdout: osd class load list = * 2023-07-02T08:59:03.532 INFO:teuthology.orchestra.run.smithi164.stdout: osd shutdown pgref assert = True 2023-07-02T08:59:03.532 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.532 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2023-07-02T08:59:03.532 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-07-02T08:59:03.532 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2023-07-02T08:59:03.533 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2023-07-02T08:59:03.533 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2023-07-02T08:59:03.533 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2023-07-02T08:59:03.533 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2023-07-02T08:59:03.533 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2023-07-02T08:59:03.533 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.533 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2023-07-02T08:59:03.533 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-07-02T08:59:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2023-07-02T08:59:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2023-07-02T08:59:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2023-07-02T08:59:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2023-07-02T08:59:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2023-07-02T08:59:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2023-07-02T08:59:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2023-07-02T08:59:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2023-07-02T08:59:03.535 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2023-07-02T08:59:03.535 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.535 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-02T08:59:03.535 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2023-07-02T08:59:03.535 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2023-07-02T08:59:03.535 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.535 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2023-07-02T08:59:03.535 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-02T08:59:03.536 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-02T08:59:03.536 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.536 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2023-07-02T08:59:03.536 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2023-07-02T08:59:03.536 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.536 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2023-07-02T08:59:03.536 INFO:teuthology.orchestra.run.smithi164.stdout: mon min osdmap epochs = 50 2023-07-02T08:59:03.536 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune interval = 2 2023-07-02T08:59:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune min = 15 2023-07-02T08:59:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune txsize = 2 2023-07-02T08:59:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: paxos service trim min = 10 2023-07-02T08:59:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on pool no app = False 2023-07-02T08:59:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default crimson = True 2023-07-02T08:59:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-07-02T08:59:03.537 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2023-07-02T08:59:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2023-07-02T08:59:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2023-07-02T08:59:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2023-07-02T08:59:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-02T08:59:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-02T08:59:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-07-02T08:59:03.538 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2023-07-02T08:59:03.538 INFO:tasks.ceph:Creating admin key on mon.a... 2023-07-02T08:59:03.539 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-07-02T08:59:03.625 INFO:tasks.ceph:Copying monmap to all nodes... 2023-07-02T08:59:03.626 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:03.626 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-07-02T08:59:03.643 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:03.644 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-07-02T08:59:03.700 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2023-07-02T08:59:03.700 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:03.701 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2023-07-02T08:59:03.701 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-02T08:59:03.776 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:03.776 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:03.834 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2023-07-02T08:59:03.834 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:03.834 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.keyring 2023-07-02T08:59:03.834 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-02T08:59:03.871 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:03.871 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:03.928 INFO:tasks.ceph:Setting up mon nodes... 2023-07-02T08:59:03.929 INFO:tasks.ceph:Setting up mgr nodes... 2023-07-02T08:59:03.929 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-07-02T08:59:04.038 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-07-02T08:59:04.041 INFO:tasks.ceph:Setting up mds nodes... 2023-07-02T08:59:04.041 INFO:tasks.ceph_client:Setting up client nodes... 2023-07-02T08:59:04.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-02T08:59:04.093 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-07-02T08:59:04.106 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-07-02T08:59:04.156 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-07-02T08:59:04.168 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-07-02T08:59:04.169 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-07-02T08:59:04.169 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-07-02T08:59:04.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-07-02T08:59:04.196 INFO:tasks.ceph:role: osd.0 2023-07-02T08:59:04.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2023-07-02T08:59:04.197 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-07-02T08:59:04.270 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T08:59:04.270 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T08:59:04.270 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T08:59:04.270 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-02T08:59:04.270 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T08:59:04.270 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2023-07-02T08:59:04.271 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T08:59:04.271 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T08:59:04.271 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T08:59:04.271 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T08:59:04.702 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2023-07-02T08:59:04.704 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2023-07-02T08:59:04.704 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-07-02T08:59:04.859 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-07-02T08:59:04.893 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-07-02T08:59:04.961 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-07-02T08:59:04.961 INFO:tasks.ceph:role: osd.1 2023-07-02T08:59:04.962 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2023-07-02T08:59:04.962 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-07-02T08:59:05.035 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T08:59:05.035 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T08:59:05.036 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T08:59:05.036 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-02T08:59:05.036 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T08:59:05.036 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2023-07-02T08:59:05.036 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T08:59:05.036 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T08:59:05.036 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T08:59:05.037 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T08:59:05.465 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2023-07-02T08:59:05.466 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2023-07-02T08:59:05.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-07-02T08:59:05.573 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-07-02T08:59:05.604 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:06.441 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2023-07-02T08:59:06.441 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:06,442 seastar - Reactor backend: epoll 2023-07-02T08:59:06.449 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:06,450 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-07-02T08:59:06.449 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:06,450 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-07-02T08:59:06.450 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:06,450 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-07-02T08:59:06.450 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:06,450 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-07-02T08:59:06.452 INFO:teuthology.orchestra.run.smithi103.stderr:==106863==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-07-02T08:59:06.458 INFO:teuthology.orchestra.run.smithi103.stderr:==106863==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffba532000; bottom 0x7f3e225ef000; size: 0x00c197f43000 (831478050816) 2023-07-02T08:59:06.458 INFO:teuthology.orchestra.run.smithi103.stderr:False positive error reports may follow 2023-07-02T08:59:06.458 INFO:teuthology.orchestra.run.smithi103.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-07-02T08:59:06.494 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:06,494 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-07-02T08:59:06.494 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:06,494 [shard 0] osd - get_early_config: set --smp 3 2023-07-02T08:59:07.084 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2023-07-02T08:59:07.084 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2023-07-02T08:59:07.084 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2023-07-02T08:59:07.085 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 InitModule 2023-07-02T08:59:07.085 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2023-07-02T08:59:07.085 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-07-02T08:59:07.155 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2023-07-02T08:59:07.156 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:07,156 seastar - Reactor backend: epoll 2023-07-02T08:59:07.158 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:07,158 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-07-02T08:59:07.158 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:07,158 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-07-02T08:59:07.158 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:07,158 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-07-02T08:59:07.158 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:07,158 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-07-02T08:59:07.160 INFO:teuthology.orchestra.run.smithi103.stderr:==106861==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-07-02T08:59:07.166 INFO:teuthology.orchestra.run.smithi103.stderr:==106861==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffba532000; bottom 0x7f3e1f4e8000; size: 0x00c19b04a000 (831529459712) 2023-07-02T08:59:07.166 INFO:teuthology.orchestra.run.smithi103.stderr:False positive error reports may follow 2023-07-02T08:59:07.166 INFO:teuthology.orchestra.run.smithi103.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-07-02T08:59:10.324 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2023-07-02T08:59:10.324 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2023-07-02T08:59:10.324 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2023-07-02T08:59:10.325 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 InitModule 2023-07-02T08:59:10.325 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2023-07-02T08:59:10.325 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-07-02T08:59:10.326 INFO:teuthology.orchestra.run.smithi103.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 334b59df-ef47-4729-a173-1813560f60f2 2023-07-02T08:59:10.384 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-02T08:59:10.411 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:11.216 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2023-07-02T08:59:11.216 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,217 seastar - Reactor backend: epoll 2023-07-02T08:59:11.224 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,225 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-07-02T08:59:11.224 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,225 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-07-02T08:59:11.225 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,225 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-07-02T08:59:11.225 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,225 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-07-02T08:59:11.226 INFO:teuthology.orchestra.run.smithi103.stderr:==107687==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-07-02T08:59:11.233 INFO:teuthology.orchestra.run.smithi103.stderr:==107687==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc6a646000; bottom 0x7f7638be8000; size: 0x008631a5e000 (576358572032) 2023-07-02T08:59:11.233 INFO:teuthology.orchestra.run.smithi103.stderr:False positive error reports may follow 2023-07-02T08:59:11.233 INFO:teuthology.orchestra.run.smithi103.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-07-02T08:59:11.268 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,269 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-07-02T08:59:11.269 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,269 [shard 0] osd - get_early_config: set --smp 3 2023-07-02T08:59:11.855 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2023-07-02T08:59:11.855 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2023-07-02T08:59:11.855 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2023-07-02T08:59:11.855 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 InitModule 2023-07-02T08:59:11.856 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2023-07-02T08:59:11.856 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-07-02T08:59:11.910 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2023-07-02T08:59:11.911 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,912 seastar - Reactor backend: epoll 2023-07-02T08:59:11.920 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,920 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-07-02T08:59:11.920 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,920 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-07-02T08:59:11.920 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,920 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-07-02T08:59:11.920 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2023-07-02 08:59:11,920 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-07-02T08:59:11.931 INFO:teuthology.orchestra.run.smithi103.stderr:==107685==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-07-02T08:59:11.936 INFO:teuthology.orchestra.run.smithi103.stderr:==107685==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc6a646000; bottom 0x7f7635ae1000; size: 0x008634b65000 (576409980928) 2023-07-02T08:59:11.937 INFO:teuthology.orchestra.run.smithi103.stderr:False positive error reports may follow 2023-07-02T08:59:11.937 INFO:teuthology.orchestra.run.smithi103.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-07-02T08:59:11.988 INFO:teuthology.orchestra.run.smithi103.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d4d 0x47b8ecd7 0x478ac1f1 0x478c6871 0x478c6d18 0x478c6e6e 0x478c72b9 0x12cef 0x3d0578f5 0x3cf4839b 0x3cfcebde 0x38ee8e27 0x38eedd5a 0x38d0fadb 0x38d2084f 0x47eabb9c 2023-07-02T08:59:11.988 INFO:teuthology.orchestra.run.smithi103.stderr:kernel callstack: 2023-07-02T08:59:15.349 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2023-07-02T08:59:15.349 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2023-07-02T08:59:15.349 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2023-07-02T08:59:15.350 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 InitModule 2023-07-02T08:59:15.350 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2023-07-02T08:59:15.350 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-07-02T08:59:15.350 INFO:teuthology.orchestra.run.smithi103.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 334b59df-ef47-4729-a173-1813560f60f2 2023-07-02T08:59:15.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-02T08:59:15.463 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-07-02T08:59:15.495 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-07-02T08:59:15.496 INFO:tasks.ceph:role: osd.2 2023-07-02T08:59:15.496 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2023-07-02T08:59:15.496 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-07-02T08:59:15.574 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T08:59:15.574 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T08:59:15.575 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T08:59:15.575 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-02T08:59:15.575 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T08:59:15.575 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-07-02T08:59:15.575 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T08:59:15.575 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T08:59:15.575 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T08:59:15.576 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T08:59:16.021 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-07-02T08:59:16.023 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-07-02T08:59:16.023 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2023-07-02T08:59:16.201 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-07-02T08:59:16.238 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-07-02T08:59:16.308 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-07-02T08:59:16.308 INFO:tasks.ceph:role: osd.3 2023-07-02T08:59:16.308 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2023-07-02T08:59:16.308 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-07-02T08:59:16.384 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T08:59:16.384 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T08:59:16.384 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T08:59:16.384 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-02T08:59:16.384 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T08:59:16.385 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-07-02T08:59:16.385 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T08:59:16.385 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T08:59:16.385 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T08:59:16.385 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T08:59:16.804 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-07-02T08:59:16.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-07-02T08:59:16.806 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2023-07-02T08:59:16.912 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-07-02T08:59:16.946 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:17.778 INFO:teuthology.orchestra.run.smithi164.stderr:WARNING: debug mode. Not for benchmarking or production 2023-07-02T08:59:17.779 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:17,779 seastar - Reactor backend: epoll 2023-07-02T08:59:17.791 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:17,791 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-07-02T08:59:17.791 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:17,791 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-07-02T08:59:17.791 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:17,791 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-07-02T08:59:17.791 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:17,791 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-07-02T08:59:17.793 INFO:teuthology.orchestra.run.smithi164.stderr:==106659==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-07-02T08:59:17.799 INFO:teuthology.orchestra.run.smithi164.stderr:==106659==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc147d9000; bottom 0x7fc88d93f000; size: 0x003386e9a000 (221306789888) 2023-07-02T08:59:17.799 INFO:teuthology.orchestra.run.smithi164.stderr:False positive error reports may follow 2023-07-02T08:59:17.799 INFO:teuthology.orchestra.run.smithi164.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-07-02T08:59:17.851 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:17,851 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-07-02T08:59:17.851 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:17,851 [shard 0] osd - get_early_config: set --smp 3 2023-07-02T08:59:18.437 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2023-07-02T08:59:18.438 INFO:teuthology.orchestra.run.smithi164.stderr:Suppressions used: 2023-07-02T08:59:18.438 INFO:teuthology.orchestra.run.smithi164.stderr: count bytes template 2023-07-02T08:59:18.438 INFO:teuthology.orchestra.run.smithi164.stderr: 1 8 InitModule 2023-07-02T08:59:18.438 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2023-07-02T08:59:18.438 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-07-02T08:59:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:WARNING: debug mode. Not for benchmarking or production 2023-07-02T08:59:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:18,504 seastar - Reactor backend: epoll 2023-07-02T08:59:18.505 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:18,506 [shard 2] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-07-02T08:59:18.505 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:18,506 [shard 2] seastar - IO queue uses 0.75ms latency goal for device 0 2023-07-02T08:59:18.506 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:18,506 [shard 2] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-07-02T08:59:18.513 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:18,514 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-07-02T08:59:18.516 INFO:teuthology.orchestra.run.smithi164.stderr:==106657==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-07-02T08:59:18.522 INFO:teuthology.orchestra.run.smithi164.stderr:==106657==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc147d9000; bottom 0x7fc88a838000; size: 0x003389fa1000 (221358198784) 2023-07-02T08:59:18.522 INFO:teuthology.orchestra.run.smithi164.stderr:False positive error reports may follow 2023-07-02T08:59:18.522 INFO:teuthology.orchestra.run.smithi164.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-07-02T08:59:21.670 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2023-07-02T08:59:21.670 INFO:teuthology.orchestra.run.smithi164.stderr:Suppressions used: 2023-07-02T08:59:21.671 INFO:teuthology.orchestra.run.smithi164.stderr: count bytes template 2023-07-02T08:59:21.671 INFO:teuthology.orchestra.run.smithi164.stderr: 1 8 InitModule 2023-07-02T08:59:21.671 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2023-07-02T08:59:21.671 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-07-02T08:59:21.671 INFO:teuthology.orchestra.run.smithi164.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 334b59df-ef47-4729-a173-1813560f60f2 2023-07-02T08:59:21.732 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-02T08:59:21.759 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:22.564 INFO:teuthology.orchestra.run.smithi164.stderr:WARNING: debug mode. Not for benchmarking or production 2023-07-02T08:59:22.565 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:22,565 seastar - Reactor backend: epoll 2023-07-02T08:59:22.573 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:22,573 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-07-02T08:59:22.573 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:22,573 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-07-02T08:59:22.573 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:22,573 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-07-02T08:59:22.573 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:22,573 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-07-02T08:59:22.575 INFO:teuthology.orchestra.run.smithi164.stderr:==107484==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-07-02T08:59:22.581 INFO:teuthology.orchestra.run.smithi164.stderr:==107484==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe77df5000; bottom 0x7fc1e76e5000; size: 0x003c90710000 (260121362432) 2023-07-02T08:59:22.581 INFO:teuthology.orchestra.run.smithi164.stderr:False positive error reports may follow 2023-07-02T08:59:22.581 INFO:teuthology.orchestra.run.smithi164.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-07-02T08:59:22.616 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:22,617 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-07-02T08:59:22.616 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:22,617 [shard 0] osd - get_early_config: set --smp 3 2023-07-02T08:59:23.182 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2023-07-02T08:59:23.183 INFO:teuthology.orchestra.run.smithi164.stderr:Suppressions used: 2023-07-02T08:59:23.183 INFO:teuthology.orchestra.run.smithi164.stderr: count bytes template 2023-07-02T08:59:23.183 INFO:teuthology.orchestra.run.smithi164.stderr: 1 8 InitModule 2023-07-02T08:59:23.183 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2023-07-02T08:59:23.183 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-07-02T08:59:23.249 INFO:teuthology.orchestra.run.smithi164.stderr:WARNING: debug mode. Not for benchmarking or production 2023-07-02T08:59:23.249 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:23,250 seastar - Reactor backend: epoll 2023-07-02T08:59:23.251 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:23,252 [shard 2] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-07-02T08:59:23.252 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:23,252 [shard 2] seastar - IO queue uses 0.75ms latency goal for device 0 2023-07-02T08:59:23.252 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:23,252 [shard 2] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-07-02T08:59:23.252 INFO:teuthology.orchestra.run.smithi164.stderr:INFO 2023-07-02 08:59:23,252 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-07-02T08:59:23.254 INFO:teuthology.orchestra.run.smithi164.stderr:==107482==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-07-02T08:59:23.260 INFO:teuthology.orchestra.run.smithi164.stderr:==107482==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe77df5000; bottom 0x7fc1e45de000; size: 0x003c93817000 (260172771328) 2023-07-02T08:59:23.260 INFO:teuthology.orchestra.run.smithi164.stderr:False positive error reports may follow 2023-07-02T08:59:23.260 INFO:teuthology.orchestra.run.smithi164.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-07-02T08:59:26.384 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2023-07-02T08:59:26.385 INFO:teuthology.orchestra.run.smithi164.stderr:Suppressions used: 2023-07-02T08:59:26.385 INFO:teuthology.orchestra.run.smithi164.stderr: count bytes template 2023-07-02T08:59:26.385 INFO:teuthology.orchestra.run.smithi164.stderr: 1 8 InitModule 2023-07-02T08:59:26.385 INFO:teuthology.orchestra.run.smithi164.stderr:----------------------------------------------------- 2023-07-02T08:59:26.385 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-07-02T08:59:26.386 INFO:teuthology.orchestra.run.smithi164.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 334b59df-ef47-4729-a173-1813560f60f2 2023-07-02T08:59:26.471 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-02T08:59:26.498 INFO:tasks.ceph:Reading keys from all nodes... 2023-07-02T08:59:26.498 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:26.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-07-02T08:59:26.531 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:26.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-07-02T08:59:26.603 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:26.603 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-07-02T08:59:26.630 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:26.630 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-07-02T08:59:26.696 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:26.697 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-07-02T08:59:26.762 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:26.763 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-07-02T08:59:26.784 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:26.784 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-07-02T08:59:26.820 INFO:tasks.ceph:Adding keys to all mons... 2023-07-02T08:59:26.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2023-07-02T08:59:26.854 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2023-07-02T08:59:26.854 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBbPKFky+scDRAAYWjdKACgRiV3YBPwAb5OdQ== 2023-07-02T08:59:26.854 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2023-07-02T08:59:26.855 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBgPKFkT5pXABAALyKhiZO9ElMtJxRscL6gbA== 2023-07-02T08:59:26.855 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2023-07-02T08:59:26.855 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBYPKFkdeBUAhAAHGS/DriDzjeoaUCF4jSObA== 2023-07-02T08:59:26.855 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2023-07-02T08:59:26.856 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBmPKFkNT1/IhAA7TlWVnJqtbmd17wTyuLD1A== 2023-07-02T08:59:26.856 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2023-07-02T08:59:26.856 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBrPKFkds+xEhAAQ2u+DE4vOiJOfgnMJz45OQ== 2023-07-02T08:59:26.857 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2023-07-02T08:59:26.857 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBYPKFkI/GYBRAAZYAdL5MTDqPhPEYP5PeoYA== 2023-07-02T08:59:26.857 INFO:teuthology.orchestra.run.smithi103.stdout:[client.1] 2023-07-02T08:59:26.857 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBYPKFkD5pXCRAA62Isg3nHIdPC/JF+UJ9rZA== 2023-07-02T08:59:26.859 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T08:59:26.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T08:59:27.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-02T08:59:27.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T08:59:27.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T08:59:27.212 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-02T08:59:27.277 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-02T08:59:27.340 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-07-02T08:59:27.341 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-07-02T08:59:27.367 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-07-02T08:59:27.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.477+0000 7f6b48f1cc40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.477+0000 7f6b48f1cc40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.492 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-07-02T08:59:27.519 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-07-02T08:59:27.576 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-07-02T08:59:27.576 INFO:tasks.ceph.mon.a:Restarting daemon 2023-07-02T08:59:27.577 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-07-02T08:59:27.618 INFO:tasks.ceph.mon.a:Started 2023-07-02T08:59:27.618 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-07-02T08:59:27.618 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-07-02T08:59:27.619 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-07-02T08:59:27.621 INFO:tasks.ceph.mgr.x:Started 2023-07-02T08:59:27.621 INFO:tasks.ceph:Setting crush tunables to default 2023-07-02T08:59:27.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd crush tunables default 2023-07-02T08:59:27.700 INFO:tasks.ceph.mon.a.smithi103.stderr:2023-07-02T08:59:27.700+0000 7f434d95ec40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.700 INFO:tasks.ceph.mon.a.smithi103.stderr:2023-07-02T08:59:27.700+0000 7f434d95ec40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.700 INFO:tasks.ceph.mon.a.smithi103.stderr:2023-07-02T08:59:27.700+0000 7f434d95ec40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:27.714+0000 7f5b939d01c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.746+0000 7fc58459c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.752+0000 7fc58459c700 1 Processor -- start 2023-07-02T08:59:27.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.752+0000 7fc58459c700 1 -- start start 2023-07-02T08:59:27.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc58459c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 0x7fc57c137830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:27.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc58459c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57c133460 con 0x7fc57c06c240 2023-07-02T08:59:27.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc58459c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57c06c650 con 0x7fc57c133050 2023-07-02T08:59:27.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc581b37700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 0x7fc57c137830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:27.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc582338700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc57c06c240 0x7fc57c132940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52458/0 (socket says 172.21.15.103:52458) 2023-07-02T08:59:27.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc582338700 1 -- 172.21.15.103:0/821921790 learned_addr learned my addr 172.21.15.103:0/821921790 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:27.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc581b37700 1 -- 172.21.15.103:0/821921790 >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 msgr2=0x7fc57c137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-02T08:59:27.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc581b37700 1 -- 172.21.15.103:0/821921790 >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 msgr2=0x7fc57c137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-02T08:59:27.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc581b37700 1 --2- 172.21.15.103:0/821921790 >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 0x7fc57c137830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-02T08:59:27.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.753+0000 7fc581b37700 1 --2- 172.21.15.103:0/821921790 >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 0x7fc57c137830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T08:59:27.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.764+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177447907 0 0) 0x7fc578001110 con 0x7fc57c06c240 2023-07-02T08:59:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.764+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc568003580 con 0x7fc57c06c240 2023-07-02T08:59:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.764+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 <== mon.0 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2406297877 0 0) 0x7fc568003580 con 0x7fc57c06c240 2023-07-02T08:59:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 msgr2=0x7fc57c137830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T08:59:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 --2- 172.21.15.103:0/821921790 >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 0x7fc57c137830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c06c830 con 0x7fc57c06c240 2023-07-02T08:59:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 <== mon.0 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (unknown 973785403 0 0) 0x7fc578001730 con 0x7fc57c06c240 2023-07-02T08:59:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 >> v1:172.21.15.103:6789/0 conn(0x7fc57c06c240 legacy=0x7fc57c132940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 --2- 172.21.15.103:0/821921790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc568003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c06c650 con 0x7fc5680039d0 2023-07-02T08:59:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc582338700 1 --2- 172.21.15.103:0/821921790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc568003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc582338700 1 -- 172.21.15.103:0/821921790 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc568003580 con 0x7fc5680039d0 2023-07-02T08:59:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc582338700 1 --2- 172.21.15.103:0/821921790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc568003e20 crc :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9b249086923ac5 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc578007e60 con 0x7fc5680039d0 2023-07-02T08:59:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc578005930 con 0x7fc5680039d0 2023-07-02T08:59:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.765+0000 7fc581336700 1 -- 172.21.15.103:0/821921790 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc578005bd0 con 0x7fc5680039d0 2023-07-02T08:59:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 1 -- 172.21.15.103:0/821921790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 msgr2=0x7fc568003e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 1 --2- 172.21.15.103:0/821921790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc568003e20 crc :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 1 -- 172.21.15.103:0/821921790 shutdown_connections 2023-07-02T08:59:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 1 --2- 172.21.15.103:0/821921790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc568003e20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 1 --2- 172.21.15.103:0/821921790 >> v2:172.21.15.103:3300/0 conn(0x7fc57c133050 0x7fc57c137830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 1 -- 172.21.15.103:0/821921790 >> 172.21.15.103:0/821921790 conn(0x7fc57c0a2530 msgr2=0x7fc57c09f830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 1 -- 172.21.15.103:0/821921790 shutdown_connections 2023-07-02T08:59:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 1 -- 172.21.15.103:0/821921790 wait complete. 2023-07-02T08:59:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.766+0000 7fc58459c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.767+0000 7fc58459c700 1 Processor -- start 2023-07-02T08:59:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.767+0000 7fc58459c700 1 -- start start 2023-07-02T08:59:27.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.768+0000 7fc58459c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc57c1210e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:27.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.768+0000 7fc58459c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c123020 con 0x7fc5680039d0 2023-07-02T08:59:27.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.768+0000 7fc582338700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc57c1210e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:27.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.768+0000 7fc582338700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc57c1210e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37070/0 (socket says 172.21.15.103:37070) 2023-07-02T08:59:27.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.768+0000 7fc582338700 1 -- 172.21.15.103:0/373751985 learned_addr learned my addr 172.21.15.103:0/373751985 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:27.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.768+0000 7fc582338700 1 -- 172.21.15.103:0/373751985 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c1225b0 con 0x7fc5680039d0 2023-07-02T08:59:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.768+0000 7fc582338700 1 --2- 172.21.15.103:0/373751985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc57c1210e0 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.769+0000 7fc572ffd700 1 -- 172.21.15.103:0/373751985 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc5780015c0 con 0x7fc5680039d0 2023-07-02T08:59:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.769+0000 7fc572ffd700 1 -- 172.21.15.103:0/373751985 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc578001720 con 0x7fc5680039d0 2023-07-02T08:59:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.769+0000 7fc572ffd700 1 -- 172.21.15.103:0/373751985 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc5780071e0 con 0x7fc5680039d0 2023-07-02T08:59:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.769+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57c120d40 con 0x7fc5680039d0 2023-07-02T08:59:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.769+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57c120950 con 0x7fc5680039d0 2023-07-02T08:59:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.769+0000 7fc572ffd700 1 -- 172.21.15.103:0/373751985 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (crc 0 0 0) 0x7fc578007390 con 0x7fc5680039d0 2023-07-02T08:59:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.769+0000 7fc572ffd700 1 -- 172.21.15.103:0/373751985 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (crc 0 0 0) 0x7fc57800c430 con 0x7fc5680039d0 2023-07-02T08:59:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.769+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57c122e90 con 0x7fc5680039d0 2023-07-02T08:59:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.772+0000 7fc572ffd700 1 -- 172.21.15.103:0/373751985 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (crc 0 0 0) 0x7fc57c133460 con 0x7fc5680039d0 2023-07-02T08:59:27.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.844+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc57c061ea0 con 0x7fc5680039d0 2023-07-02T08:59:27.846 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.846+0000 7fc572ffd700 1 -- 172.21.15.103:0/373751985 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (crc 0 0 0) 0x7fc578016420 con 0x7fc5680039d0 2023-07-02T08:59:27.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.847+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 msgr2=0x7fc57c1210e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:27.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.847+0000 7fc58459c700 1 --2- 172.21.15.103:0/373751985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc57c1210e0 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:27.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.847+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 shutdown_connections 2023-07-02T08:59:27.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.847+0000 7fc58459c700 1 --2- 172.21.15.103:0/373751985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5680039d0 0x7fc57c1210e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:27.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.847+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 >> 172.21.15.103:0/373751985 conn(0x7fc57c0a2530 msgr2=0x7fc57c070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:27.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.847+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 shutdown_connections 2023-07-02T08:59:27.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.847+0000 7fc58459c700 1 -- 172.21.15.103:0/373751985 wait complete. 2023-07-02T08:59:27.849 INFO:teuthology.orchestra.run.smithi103.stderr:adjusted tunables profile to default 2023-07-02T08:59:27.859 INFO:tasks.ceph:check_enable_crimson: True 2023-07-02T08:59:27.859 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-07-02T08:59:27.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-07-02T08:59:27.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:27.918+0000 7f5b939d01c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:27.918+0000 7f5b939d01c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.981+0000 7f9fb634b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.986+0000 7f9fb634b700 1 Processor -- start 2023-07-02T08:59:27.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.986+0000 7f9fb634b700 1 -- start start 2023-07-02T08:59:27.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9fb634b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9fb009a220 0x7f9fb01348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9fb634b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb009cea0 con 0x7f9fb006d6e0 2023-07-02T08:59:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9fb634b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb006d490 con 0x7f9fb009a220 2023-07-02T08:59:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faf7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9fb009a220 0x7f9fb01348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faf7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9fb009a220 0x7f9fb01348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37074/0 (socket says 172.21.15.103:37074) 2023-07-02T08:59:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faffff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9fb006d6e0 0x7f9fb0099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52464/0 (socket says 172.21.15.103:52464) 2023-07-02T08:59:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faf7fe700 1 -- 172.21.15.103:0/172592111 learned_addr learned my addr 172.21.15.103:0/172592111 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faf7fe700 1 -- 172.21.15.103:0/172592111 >> v1:172.21.15.103:6789/0 conn(0x7f9fb006d6e0 legacy=0x7f9fb0099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faf7fe700 1 -- 172.21.15.103:0/172592111 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb006dc80 con 0x7f9fb009a220 2023-07-02T08:59:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faf7fe700 1 --2- 172.21.15.103:0/172592111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb009a220 0x7f9fb01348e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342aca4c79e4586c server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faeffd700 1 -- 172.21.15.103:0/172592111 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9fa0009d90 con 0x7f9fb009a220 2023-07-02T08:59:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faeffd700 1 -- 172.21.15.103:0/172592111 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9fa000e230 con 0x7f9fb009a220 2023-07-02T08:59:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.987+0000 7f9faeffd700 1 -- 172.21.15.103:0/172592111 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9fa000e4d0 con 0x7f9fb009a220 2023-07-02T08:59:27.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.988+0000 7f9fb634b700 1 -- 172.21.15.103:0/172592111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb009a220 msgr2=0x7f9fb01348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:27.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.988+0000 7f9fb634b700 1 --2- 172.21.15.103:0/172592111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb009a220 0x7f9fb01348e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:27.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.988+0000 7f9fb634b700 1 -- 172.21.15.103:0/172592111 shutdown_connections 2023-07-02T08:59:27.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.988+0000 7f9fb634b700 1 --2- 172.21.15.103:0/172592111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb009a220 0x7f9fb01348e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:27.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.988+0000 7f9fb634b700 1 -- 172.21.15.103:0/172592111 >> 172.21.15.103:0/172592111 conn(0x7f9fb009d810 msgr2=0x7f9fb009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:27.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.988+0000 7f9fb634b700 1 -- 172.21.15.103:0/172592111 shutdown_connections 2023-07-02T08:59:27.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.988+0000 7f9fb634b700 1 -- 172.21.15.103:0/172592111 wait complete. 2023-07-02T08:59:27.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.988+0000 7f9fb634b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:27.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.989+0000 7f9fb634b700 1 Processor -- start 2023-07-02T08:59:27.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.989+0000 7f9fb634b700 1 -- start start 2023-07-02T08:59:27.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.990+0000 7f9fb634b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb006d6e0 0x7f9fb01c3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:27.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.990+0000 7f9fb634b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb011de10 con 0x7f9fb006d6e0 2023-07-02T08:59:27.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.990+0000 7f9faffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb006d6e0 0x7f9fb01c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:27.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.990+0000 7f9faffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb006d6e0 0x7f9fb01c3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37090/0 (socket says 172.21.15.103:37090) 2023-07-02T08:59:27.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.990+0000 7f9faffff700 1 -- 172.21.15.103:0/133147062 learned_addr learned my addr 172.21.15.103:0/133147062 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:27.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.990+0000 7f9faffff700 1 -- 172.21.15.103:0/133147062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb011f650 con 0x7f9fb006d6e0 2023-07-02T08:59:27.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.991+0000 7f9faffff700 1 --2- 172.21.15.103:0/133147062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb006d6e0 0x7f9fb01c3930 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:27.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.991+0000 7f9facff9700 1 -- 172.21.15.103:0/133147062 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9fa40059a0 con 0x7f9fb006d6e0 2023-07-02T08:59:27.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.991+0000 7f9facff9700 1 -- 172.21.15.103:0/133147062 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9fa4005b00 con 0x7f9fb006d6e0 2023-07-02T08:59:27.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.991+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb011db00 con 0x7f9fb006d6e0 2023-07-02T08:59:27.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.991+0000 7f9facff9700 1 -- 172.21.15.103:0/133147062 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9fa4005da0 con 0x7f9fb006d6e0 2023-07-02T08:59:27.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.991+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb0120f20 con 0x7f9fb006d6e0 2023-07-02T08:59:27.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.991+0000 7f9facff9700 1 -- 172.21.15.103:0/133147062 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (crc 0 0 0) 0x7f9fa40073b0 con 0x7f9fb006d6e0 2023-07-02T08:59:27.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.991+0000 7f9facff9700 1 -- 172.21.15.103:0/133147062 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (crc 0 0 0) 0x7f9fa400f5c0 con 0x7f9fb006d6e0 2023-07-02T08:59:27.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.992+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb0061ea0 con 0x7f9fb006d6e0 2023-07-02T08:59:27.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:27.994+0000 7f9facff9700 1 -- 172.21.15.103:0/133147062 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (crc 0 0 0) 0x7f9fa400c850 con 0x7f9fb006d6e0 2023-07-02T08:59:28.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:28.054+0000 7f5b939d01c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-02T08:59:28.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.077+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9fb0073ca0 con 0x7f9fb006d6e0 2023-07-02T08:59:28.077 INFO:tasks.ceph.mon.a.smithi103.stderr:2023-07-02T08:59:28.077+0000 7f4340c1a700 -1 WARNING: experimental feature 'crimson' is enabled 2023-07-02T08:59:28.078 INFO:tasks.ceph.mon.a.smithi103.stderr:Please be aware that this feature is experimental, untested, 2023-07-02T08:59:28.078 INFO:tasks.ceph.mon.a.smithi103.stderr:unsupported, and may result in data corruption, data loss, 2023-07-02T08:59:28.078 INFO:tasks.ceph.mon.a.smithi103.stderr:and/or irreparable damage to your cluster. Do not use 2023-07-02T08:59:28.078 INFO:tasks.ceph.mon.a.smithi103.stderr:feature with important data. 2023-07-02T08:59:28.078 INFO:tasks.ceph.mon.a.smithi103.stderr: 2023-07-02T08:59:28.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:28.226+0000 7f5b939d01c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-02T08:59:28.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:28.628+0000 7f5b939d01c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-02T08:59:28.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.847+0000 7f9facff9700 1 -- 172.21.15.103:0/133147062 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true}]=0 v3) v1 ==== 99+0+0 (crc 0 0 0) 0x7f9fa400cac0 con 0x7f9fb006d6e0 2023-07-02T08:59:28.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.848+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb006d6e0 msgr2=0x7f9fb01c3930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:28.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.848+0000 7f9fb634b700 1 --2- 172.21.15.103:0/133147062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb006d6e0 0x7f9fb01c3930 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:28.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.848+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 shutdown_connections 2023-07-02T08:59:28.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.848+0000 7f9fb634b700 1 --2- 172.21.15.103:0/133147062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb006d6e0 0x7f9fb01c3930 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:28.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.848+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 >> 172.21.15.103:0/133147062 conn(0x7f9fb009d810 msgr2=0x7f9fb0117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:28.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.848+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 shutdown_connections 2023-07-02T08:59:28.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:28.848+0000 7f9fb634b700 1 -- 172.21.15.103:0/133147062 wait complete. 2023-07-02T08:59:28.862 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-07-02T08:59:28.862 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:28.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-07-02T08:59:28.892 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-07-02T08:59:28.892 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-07-02T08:59:28.960 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:28.960 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-07-02T08:59:28.988 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-07-02T08:59:28.988 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-07-02T08:59:29.057 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 9838c66d-5b32-4a3c-b090-4d92ba53e4ba 0 2023-07-02T08:59:29.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:29.129+0000 7f5b939d01c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-02T08:59:29.226 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.220+0000 7f334d11b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:29.226 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f334d11b700 1 Processor -- start 2023-07-02T08:59:29.227 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f334d11b700 1 -- start start 2023-07-02T08:59:29.227 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f334d11b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3348117cf0 0x7f334811a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:29.227 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f334d11b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334811a770 con 0x7f3348115080 2023-07-02T08:59:29.227 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f334d11b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334811b930 con 0x7f3348117cf0 2023-07-02T08:59:29.229 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f334659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3348117cf0 0x7f334811a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:29.229 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f3346d9d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3348115080 0x7f3348115490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.164:56464/0 (socket says 172.21.15.164:56464) 2023-07-02T08:59:29.229 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f3346d9d700 1 -- 172.21.15.164:0/1448362607 learned_addr learned my addr 172.21.15.164:0/1448362607 (peer_addr_for_me v1:172.21.15.164:0/0) 2023-07-02T08:59:29.229 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.226+0000 7f334659c700 1 --2- 172.21.15.164:0/1448362607 >> v2:172.21.15.103:3300/0 conn(0x7f3348117cf0 0x7f334811a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.164:42652/0 (socket says 172.21.15.164:42652) 2023-07-02T08:59:29.229 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.227+0000 7f334659c700 1 -- 172.21.15.164:0/1448362607 >> v1:172.21.15.103:6789/0 conn(0x7f3348115080 legacy=0x7f3348115490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.229 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.227+0000 7f334659c700 1 -- 172.21.15.164:0/1448362607 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334811baa0 con 0x7f3348117cf0 2023-07-02T08:59:29.230 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.227+0000 7f3345d9b700 1 -- 172.21.15.164:0/1448362607 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582309046 0 0) 0x7f334811a770 con 0x7f3348115080 2023-07-02T08:59:29.230 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.227+0000 7f334659c700 1 --2- 172.21.15.164:0/1448362607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348117cf0 0x7f334811a110 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e385de0745554a2 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:29.230 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.227+0000 7f3345d9b700 1 -- 172.21.15.164:0/1448362607 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f333c009cd0 con 0x7f3348117cf0 2023-07-02T08:59:29.230 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.227+0000 7f3345d9b700 1 -- 172.21.15.164:0/1448362607 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f333c009e30 con 0x7f3348117cf0 2023-07-02T08:59:29.230 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.227+0000 7f3345d9b700 1 -- 172.21.15.164:0/1448362607 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f333c00e360 con 0x7f3348117cf0 2023-07-02T08:59:29.230 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.228+0000 7f334d11b700 1 -- 172.21.15.164:0/1448362607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348117cf0 msgr2=0x7f334811a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.231 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.228+0000 7f334d11b700 1 --2- 172.21.15.164:0/1448362607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348117cf0 0x7f334811a110 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.231 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.228+0000 7f334d11b700 1 -- 172.21.15.164:0/1448362607 shutdown_connections 2023-07-02T08:59:29.231 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.228+0000 7f334d11b700 1 --2- 172.21.15.164:0/1448362607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348117cf0 0x7f334811a110 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.231 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.228+0000 7f334d11b700 1 -- 172.21.15.164:0/1448362607 >> 172.21.15.164:0/1448362607 conn(0x7f334806e920 msgr2=0x7f3348070d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:29.231 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.228+0000 7f334d11b700 1 -- 172.21.15.164:0/1448362607 shutdown_connections 2023-07-02T08:59:29.231 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.228+0000 7f334d11b700 1 -- 172.21.15.164:0/1448362607 wait complete. 2023-07-02T08:59:29.231 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.228+0000 7f334d11b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:29.233 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.229+0000 7f334d11b700 1 Processor -- start 2023-07-02T08:59:29.233 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.229+0000 7f334d11b700 1 -- start start 2023-07-02T08:59:29.233 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.229+0000 7f334d11b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348115080 0x7f33481a9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:29.233 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.229+0000 7f334d11b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33481a9700 con 0x7f3348115080 2023-07-02T08:59:29.233 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.229+0000 7f3346d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348115080 0x7f33481a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:29.233 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.229+0000 7f3346d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348115080 0x7f33481a9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.164:42654/0 (socket says 172.21.15.164:42654) 2023-07-02T08:59:29.234 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.229+0000 7f3346d9d700 1 -- 172.21.15.164:0/1731435846 learned_addr learned my addr 172.21.15.164:0/1731435846 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-07-02T08:59:29.234 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f3346d9d700 1 -- 172.21.15.164:0/1731435846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33481a9840 con 0x7f3348115080 2023-07-02T08:59:29.234 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f3346d9d700 1 --2- 172.21.15.164:0/1731435846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348115080 0x7f33481a9130 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:29.234 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f332f7fe700 1 -- 172.21.15.164:0/1731435846 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3338004b80 con 0x7f3348115080 2023-07-02T08:59:29.234 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f332f7fe700 1 -- 172.21.15.164:0/1731435846 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3338004ce0 con 0x7f3348115080 2023-07-02T08:59:29.234 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33481a9a90 con 0x7f3348115080 2023-07-02T08:59:29.234 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f332f7fe700 1 -- 172.21.15.164:0/1731435846 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f333800c5f0 con 0x7f3348115080 2023-07-02T08:59:29.235 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33481a34e0 con 0x7f3348115080 2023-07-02T08:59:29.235 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f332f7fe700 1 -- 172.21.15.164:0/1731435846 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (crc 0 0 0) 0x7f333800ca50 con 0x7f3348115080 2023-07-02T08:59:29.235 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.230+0000 7f332f7fe700 1 -- 172.21.15.164:0/1731435846 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 747+0+0 (crc 0 0 0) 0x7f333800d150 con 0x7f3348115080 2023-07-02T08:59:29.235 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.231+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3348068670 con 0x7f3348115080 2023-07-02T08:59:29.235 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.233+0000 7f332f7fe700 1 -- 172.21.15.164:0/1731435846 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (crc 0 0 0) 0x7f333800ee70 con 0x7f3348115080 2023-07-02T08:59:29.315 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.315+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9838c66d-5b32-4a3c-b090-4d92ba53e4ba", "id": 0} v 0) v1 -- 0x7f3348117e30 con 0x7f3348115080 2023-07-02T08:59:29.318 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.318+0000 7f332f7fe700 1 -- 172.21.15.164:0/1731435846 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9838c66d-5b32-4a3c-b090-4d92ba53e4ba", "id": 0}]=0 v4) v1 ==== 112+0+2 (crc 0 0 0) 0x7f333800e290 con 0x7f3348115080 2023-07-02T08:59:29.319 INFO:teuthology.orchestra.run.smithi164.stdout:0 2023-07-02T08:59:29.319 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.319+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348115080 msgr2=0x7f33481a9130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.319+0000 7f334d11b700 1 --2- 172.21.15.164:0/1731435846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348115080 0x7f33481a9130 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.319+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 shutdown_connections 2023-07-02T08:59:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.319+0000 7f334d11b700 1 --2- 172.21.15.164:0/1731435846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3348115080 0x7f33481a9130 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.319+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 >> 172.21.15.164:0/1731435846 conn(0x7f334806e920 msgr2=0x7f3348118f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.319+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 shutdown_connections 2023-07-02T08:59:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.319+0000 7f334d11b700 1 -- 172.21.15.164:0/1731435846 wait complete. 2023-07-02T08:59:29.331 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new fc54ba3d-23a0-488e-9e81-8d60a083b6eb 1 2023-07-02T08:59:29.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:29.332+0000 7f5b939d01c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-02T08:59:29.461 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.455+0000 7fb4ae0db700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:29.461 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.461+0000 7fb4ae0db700 1 Processor -- start 2023-07-02T08:59:29.462 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.461+0000 7fb4ae0db700 1 -- start start 2023-07-02T08:59:29.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.461+0000 7fb4ae0db700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb4a8119ed0 0x7fb4a811c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:29.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.461+0000 7fb4ae0db700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a811c8f0 con 0x7fb4a8117210 2023-07-02T08:59:29.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.461+0000 7fb4ae0db700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a811dab0 con 0x7fb4a8119ed0 2023-07-02T08:59:29.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.462+0000 7fb4a6ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb4a8119ed0 0x7fb4a811c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:29.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.462+0000 7fb4a77fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb4a8117210 0x7fb4a8117620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.164:56480/0 (socket says 172.21.15.164:56480) 2023-07-02T08:59:29.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.462+0000 7fb4a77fe700 1 -- 172.21.15.164:0/3895523991 learned_addr learned my addr 172.21.15.164:0/3895523991 (peer_addr_for_me v1:172.21.15.164:0/0) 2023-07-02T08:59:29.464 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.462+0000 7fb4a6ffd700 1 -- 172.21.15.164:0/3895523991 >> v1:172.21.15.103:6789/0 conn(0x7fb4a8117210 legacy=0x7fb4a8117620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.464 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.462+0000 7fb4a6ffd700 1 -- 172.21.15.164:0/3895523991 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a811dc20 con 0x7fb4a8119ed0 2023-07-02T08:59:29.464 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.462+0000 7fb4a6ffd700 1 --2- 172.21.15.164:0/3895523991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8119ed0 0x7fb4a811c2f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd40fdfc76c57666 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:29.464 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.462+0000 7fb4a67fc700 1 -- 172.21.15.164:0/3895523991 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb49c009cd0 con 0x7fb4a8119ed0 2023-07-02T08:59:29.464 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.462+0000 7fb4a67fc700 1 -- 172.21.15.164:0/3895523991 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb49c009e30 con 0x7fb4a8119ed0 2023-07-02T08:59:29.464 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.463+0000 7fb4a67fc700 1 -- 172.21.15.164:0/3895523991 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb49c00e360 con 0x7fb4a8119ed0 2023-07-02T08:59:29.466 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.463+0000 7fb4ae0db700 1 -- 172.21.15.164:0/3895523991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8119ed0 msgr2=0x7fb4a811c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.466 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.463+0000 7fb4ae0db700 1 --2- 172.21.15.164:0/3895523991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8119ed0 0x7fb4a811c2f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.467 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.464+0000 7fb4ae0db700 1 -- 172.21.15.164:0/3895523991 shutdown_connections 2023-07-02T08:59:29.467 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.464+0000 7fb4ae0db700 1 --2- 172.21.15.164:0/3895523991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8119ed0 0x7fb4a811c2f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.467 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.464+0000 7fb4ae0db700 1 -- 172.21.15.164:0/3895523991 >> 172.21.15.164:0/3895523991 conn(0x7fb4a806ce60 msgr2=0x7fb4a806d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:29.467 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.464+0000 7fb4ae0db700 1 -- 172.21.15.164:0/3895523991 shutdown_connections 2023-07-02T08:59:29.467 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.464+0000 7fb4ae0db700 1 -- 172.21.15.164:0/3895523991 wait complete. 2023-07-02T08:59:29.467 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.464+0000 7fb4ae0db700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:29.467 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.464+0000 7fb4ae0db700 1 Processor -- start 2023-07-02T08:59:29.467 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.465+0000 7fb4ae0db700 1 -- start start 2023-07-02T08:59:29.468 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.465+0000 7fb4ae0db700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8117210 0x7fb4a81a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:29.468 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.465+0000 7fb4ae0db700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81a6000 con 0x7fb4a8117210 2023-07-02T08:59:29.468 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.465+0000 7fb4a77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8117210 0x7fb4a81a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:29.468 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.465+0000 7fb4a77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8117210 0x7fb4a81a5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.164:42664/0 (socket says 172.21.15.164:42664) 2023-07-02T08:59:29.468 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.465+0000 7fb4a77fe700 1 -- 172.21.15.164:0/56646705 learned_addr learned my addr 172.21.15.164:0/56646705 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-07-02T08:59:29.468 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.465+0000 7fb4a77fe700 1 -- 172.21.15.164:0/56646705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a81a6140 con 0x7fb4a8117210 2023-07-02T08:59:29.468 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.466+0000 7fb4a77fe700 1 --2- 172.21.15.164:0/56646705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8117210 0x7fb4a81a5a30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:29.469 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.466+0000 7fb48ffff700 1 -- 172.21.15.164:0/56646705 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb4980070b0 con 0x7fb4a8117210 2023-07-02T08:59:29.469 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.466+0000 7fb48ffff700 1 -- 172.21.15.164:0/56646705 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb498007210 con 0x7fb4a8117210 2023-07-02T08:59:29.469 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.466+0000 7fb48ffff700 1 -- 172.21.15.164:0/56646705 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb4980074b0 con 0x7fb4a8117210 2023-07-02T08:59:29.469 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.466+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a81a63f0 con 0x7fb4a8117210 2023-07-02T08:59:29.469 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.466+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a81aa2e0 con 0x7fb4a8117210 2023-07-02T08:59:29.469 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.466+0000 7fb48ffff700 1 -- 172.21.15.164:0/56646705 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (crc 0 0 0) 0x7fb4980079a0 con 0x7fb4a8117210 2023-07-02T08:59:29.469 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.466+0000 7fb48ffff700 1 -- 172.21.15.164:0/56646705 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 862+0+0 (crc 0 0 0) 0x7fb49800e580 con 0x7fb4a8117210 2023-07-02T08:59:29.470 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.467+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a8068670 con 0x7fb4a8117210 2023-07-02T08:59:29.470 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.469+0000 7fb48ffff700 1 -- 172.21.15.164:0/56646705 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (crc 0 0 0) 0x7fb49800f320 con 0x7fb4a8117210 2023-07-02T08:59:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.570+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fc54ba3d-23a0-488e-9e81-8d60a083b6eb", "id": 1} v 0) v1 -- 0x7fb4a8119f30 con 0x7fb4a8117210 2023-07-02T08:59:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.573+0000 7fb48ffff700 1 -- 172.21.15.164:0/56646705 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc54ba3d-23a0-488e-9e81-8d60a083b6eb", "id": 1}]=0 v5) v1 ==== 112+0+2 (crc 0 0 0) 0x7fb49800f540 con 0x7fb4a8117210 2023-07-02T08:59:29.574 INFO:teuthology.orchestra.run.smithi164.stdout:1 2023-07-02T08:59:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.574+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8117210 msgr2=0x7fb4a81a5a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.574+0000 7fb4ae0db700 1 --2- 172.21.15.164:0/56646705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8117210 0x7fb4a81a5a30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.574+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 shutdown_connections 2023-07-02T08:59:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.574+0000 7fb4ae0db700 1 --2- 172.21.15.164:0/56646705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a8117210 0x7fb4a81a5a30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.574+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 >> 172.21.15.164:0/56646705 conn(0x7fb4a806ce60 msgr2=0x7fb4a810fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.574+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 shutdown_connections 2023-07-02T08:59:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.574+0000 7fb4ae0db700 1 -- 172.21.15.164:0/56646705 wait complete. 2023-07-02T08:59:29.587 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 90ee74d5-14e0-4681-a539-dbb849475cec 2 2023-07-02T08:59:29.716 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.710+0000 7fb5d73fd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:29.717 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.717+0000 7fb5d73fd700 1 Processor -- start 2023-07-02T08:59:29.717 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.717+0000 7fb5d73fd700 1 -- start start 2023-07-02T08:59:29.718 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.717+0000 7fb5d73fd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb5d0119ed0 0x7fb5d011c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:29.718 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.717+0000 7fb5d73fd700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d011c8f0 con 0x7fb5d0117210 2023-07-02T08:59:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.717+0000 7fb5d73fd700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d011dab0 con 0x7fb5d0119ed0 2023-07-02T08:59:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.717+0000 7fb5d4998700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb5d0119ed0 0x7fb5d011c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.717+0000 7fb5d5199700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb5d0117210 0x7fb5d0117620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.164:56494/0 (socket says 172.21.15.164:56494) 2023-07-02T08:59:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.717+0000 7fb5d5199700 1 -- 172.21.15.164:0/4111083379 learned_addr learned my addr 172.21.15.164:0/4111083379 (peer_addr_for_me v1:172.21.15.164:0/0) 2023-07-02T08:59:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.718+0000 7fb5d4998700 1 -- 172.21.15.164:0/4111083379 >> v1:172.21.15.103:6789/0 conn(0x7fb5d0117210 legacy=0x7fb5d0117620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.718+0000 7fb5d4998700 1 -- 172.21.15.164:0/4111083379 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d011dc20 con 0x7fb5d0119ed0 2023-07-02T08:59:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.718+0000 7fb5d4998700 1 --2- 172.21.15.164:0/4111083379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0119ed0 0x7fb5d011c2f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde0029d841a29fb server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:29.720 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.718+0000 7fb5c7fff700 1 -- 172.21.15.164:0/4111083379 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb5c0009cd0 con 0x7fb5d0119ed0 2023-07-02T08:59:29.720 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.718+0000 7fb5c7fff700 1 -- 172.21.15.164:0/4111083379 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb5c0009e30 con 0x7fb5d0119ed0 2023-07-02T08:59:29.720 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.718+0000 7fb5c7fff700 1 -- 172.21.15.164:0/4111083379 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb5c000e360 con 0x7fb5d0119ed0 2023-07-02T08:59:29.722 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.719+0000 7fb5d73fd700 1 -- 172.21.15.164:0/4111083379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0119ed0 msgr2=0x7fb5d011c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.722 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.719+0000 7fb5d73fd700 1 --2- 172.21.15.164:0/4111083379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0119ed0 0x7fb5d011c2f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.722 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.719+0000 7fb5d73fd700 1 -- 172.21.15.164:0/4111083379 shutdown_connections 2023-07-02T08:59:29.722 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.719+0000 7fb5d73fd700 1 --2- 172.21.15.164:0/4111083379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0119ed0 0x7fb5d011c2f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.722 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.719+0000 7fb5d73fd700 1 -- 172.21.15.164:0/4111083379 >> 172.21.15.164:0/4111083379 conn(0x7fb5d006ce60 msgr2=0x7fb5d006d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:29.722 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.719+0000 7fb5d73fd700 1 -- 172.21.15.164:0/4111083379 shutdown_connections 2023-07-02T08:59:29.723 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.719+0000 7fb5d73fd700 1 -- 172.21.15.164:0/4111083379 wait complete. 2023-07-02T08:59:29.723 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.719+0000 7fb5d73fd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:29.723 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.720+0000 7fb5d73fd700 1 Processor -- start 2023-07-02T08:59:29.723 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.720+0000 7fb5d73fd700 1 -- start start 2023-07-02T08:59:29.723 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.720+0000 7fb5d73fd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0117210 0x7fb5d01a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:29.723 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.720+0000 7fb5d73fd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d01a6000 con 0x7fb5d0117210 2023-07-02T08:59:29.723 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.720+0000 7fb5d5199700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0117210 0x7fb5d01a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:29.724 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.720+0000 7fb5d5199700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0117210 0x7fb5d01a5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.164:42678/0 (socket says 172.21.15.164:42678) 2023-07-02T08:59:29.724 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.720+0000 7fb5d5199700 1 -- 172.21.15.164:0/150873070 learned_addr learned my addr 172.21.15.164:0/150873070 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-07-02T08:59:29.724 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5d5199700 1 -- 172.21.15.164:0/150873070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d01a6140 con 0x7fb5d0117210 2023-07-02T08:59:29.724 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5d5199700 1 --2- 172.21.15.164:0/150873070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0117210 0x7fb5d01a5a30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:29.724 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5c5ffb700 1 -- 172.21.15.164:0/150873070 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb5cc005a60 con 0x7fb5d0117210 2023-07-02T08:59:29.724 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5c5ffb700 1 -- 172.21.15.164:0/150873070 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb5cc005bc0 con 0x7fb5d0117210 2023-07-02T08:59:29.724 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d01a63f0 con 0x7fb5d0117210 2023-07-02T08:59:29.724 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5c5ffb700 1 -- 172.21.15.164:0/150873070 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb5cc005e60 con 0x7fb5d0117210 2023-07-02T08:59:29.725 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5c5ffb700 1 -- 172.21.15.164:0/150873070 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (crc 0 0 0) 0x7fb5cc00cba0 con 0x7fb5d0117210 2023-07-02T08:59:29.725 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d01aa2e0 con 0x7fb5d0117210 2023-07-02T08:59:29.725 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.721+0000 7fb5c5ffb700 1 -- 172.21.15.164:0/150873070 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 977+0+0 (crc 0 0 0) 0x7fb5cc007540 con 0x7fb5d0117210 2023-07-02T08:59:29.725 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.722+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d0068670 con 0x7fb5d0117210 2023-07-02T08:59:29.725 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.725+0000 7fb5c5ffb700 1 -- 172.21.15.164:0/150873070 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (crc 0 0 0) 0x7fb5cc007100 con 0x7fb5d0117210 2023-07-02T08:59:29.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:29.757+0000 7f5b939d01c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-02T08:59:29.805 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.805+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "90ee74d5-14e0-4681-a539-dbb849475cec", "id": 2} v 0) v1 -- 0x7fb5d0119f30 con 0x7fb5d0117210 2023-07-02T08:59:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.808+0000 7fb5c5ffb700 1 -- 172.21.15.164:0/150873070 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90ee74d5-14e0-4681-a539-dbb849475cec", "id": 2}]=0 v6) v1 ==== 112+0+2 (crc 0 0 0) 0x7fb5cc007320 con 0x7fb5d0117210 2023-07-02T08:59:29.809 INFO:teuthology.orchestra.run.smithi164.stdout:2 2023-07-02T08:59:29.810 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.809+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0117210 msgr2=0x7fb5d01a5a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.810 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.809+0000 7fb5d73fd700 1 --2- 172.21.15.164:0/150873070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0117210 0x7fb5d01a5a30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.810 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.809+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 shutdown_connections 2023-07-02T08:59:29.810 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.809+0000 7fb5d73fd700 1 --2- 172.21.15.164:0/150873070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5d0117210 0x7fb5d01a5a30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.810 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.809+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 >> 172.21.15.164:0/150873070 conn(0x7fb5d006ce60 msgr2=0x7fb5d010fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:29.810 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.809+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 shutdown_connections 2023-07-02T08:59:29.811 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.809+0000 7fb5d73fd700 1 -- 172.21.15.164:0/150873070 wait complete. 2023-07-02T08:59:29.821 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 647cd0e3-4f38-4dc4-aeb7-d4d2784493da 3 2023-07-02T08:59:29.948 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.943+0000 7fe54dd84700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:29.949 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.949+0000 7fe54dd84700 1 Processor -- start 2023-07-02T08:59:29.949 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.949+0000 7fe54dd84700 1 -- start start 2023-07-02T08:59:29.950 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.949+0000 7fe54dd84700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe54806bef0 0x7fe54806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:29.950 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.949+0000 7fe54dd84700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54811c9e0 con 0x7fe54806c840 2023-07-02T08:59:29.950 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.949+0000 7fe54dd84700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54811dba0 con 0x7fe54806bef0 2023-07-02T08:59:29.951 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.949+0000 7fe5477fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe54806bef0 0x7fe54806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:29.951 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe546ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe54806c840 0x7fe54811b1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.164:56496/0 (socket says 172.21.15.164:56496) 2023-07-02T08:59:29.952 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe5477fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe54806bef0 0x7fe54806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.164:42686/0 (socket says 172.21.15.164:42686) 2023-07-02T08:59:29.952 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe546ffd700 1 -- 172.21.15.164:0/1338044245 learned_addr learned my addr 172.21.15.164:0/1338044245 (peer_addr_for_me v1:172.21.15.164:0/0) 2023-07-02T08:59:29.952 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe5477fe700 1 -- 172.21.15.164:0/1338044245 >> v1:172.21.15.103:6789/0 conn(0x7fe54806c840 legacy=0x7fe54811b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.952 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe5477fe700 1 -- 172.21.15.164:0/1338044245 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54811dd10 con 0x7fe54806bef0 2023-07-02T08:59:29.952 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe5477fe700 1 --2- 172.21.15.164:0/1338044245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe54806c300 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742ea56bf423d4a3 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:29.952 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe5467fc700 1 -- 172.21.15.164:0/1338044245 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe538009cd0 con 0x7fe54806bef0 2023-07-02T08:59:29.952 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe5467fc700 1 -- 172.21.15.164:0/1338044245 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe538009e30 con 0x7fe54806bef0 2023-07-02T08:59:29.953 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.950+0000 7fe5467fc700 1 -- 172.21.15.164:0/1338044245 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe53800e360 con 0x7fe54806bef0 2023-07-02T08:59:29.953 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.951+0000 7fe54dd84700 1 -- 172.21.15.164:0/1338044245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 msgr2=0x7fe54806c300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:29.953 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.951+0000 7fe54dd84700 1 --2- 172.21.15.164:0/1338044245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe54806c300 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.953 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.951+0000 7fe54dd84700 1 -- 172.21.15.164:0/1338044245 shutdown_connections 2023-07-02T08:59:29.953 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.951+0000 7fe54dd84700 1 --2- 172.21.15.164:0/1338044245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe54806c300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:29.953 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.951+0000 7fe54dd84700 1 -- 172.21.15.164:0/1338044245 >> 172.21.15.164:0/1338044245 conn(0x7fe54806f720 msgr2=0x7fe548071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:29.953 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.951+0000 7fe54dd84700 1 -- 172.21.15.164:0/1338044245 shutdown_connections 2023-07-02T08:59:29.954 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.951+0000 7fe54dd84700 1 -- 172.21.15.164:0/1338044245 wait complete. 2023-07-02T08:59:29.954 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.951+0000 7fe54dd84700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:29.955 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.952+0000 7fe54dd84700 1 Processor -- start 2023-07-02T08:59:29.955 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.952+0000 7fe54dd84700 1 -- start start 2023-07-02T08:59:29.955 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.952+0000 7fe54dd84700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe5481a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:29.955 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.952+0000 7fe54dd84700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481a6070 con 0x7fe54806bef0 2023-07-02T08:59:29.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.952+0000 7fe5477fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe5481a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:29.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.952+0000 7fe5477fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe5481a5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.164:42694/0 (socket says 172.21.15.164:42694) 2023-07-02T08:59:29.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.952+0000 7fe5477fe700 1 -- 172.21.15.164:0/1041874015 learned_addr learned my addr 172.21.15.164:0/1041874015 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-07-02T08:59:29.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe5477fe700 1 -- 172.21.15.164:0/1041874015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5481a61b0 con 0x7fe54806bef0 2023-07-02T08:59:29.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe5477fe700 1 --2- 172.21.15.164:0/1041874015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe5481a5b30 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:29.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe52ffff700 1 -- 172.21.15.164:0/1041874015 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe538013530 con 0x7fe54806bef0 2023-07-02T08:59:29.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe52ffff700 1 -- 172.21.15.164:0/1041874015 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe538013690 con 0x7fe54806bef0 2023-07-02T08:59:29.957 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe52ffff700 1 -- 172.21.15.164:0/1041874015 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe538013930 con 0x7fe54806bef0 2023-07-02T08:59:29.957 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5481a6400 con 0x7fe54806bef0 2023-07-02T08:59:29.957 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5481aa2f0 con 0x7fe54806bef0 2023-07-02T08:59:29.957 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe52ffff700 1 -- 172.21.15.164:0/1041874015 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (crc 0 0 0) 0x7fe5380123f0 con 0x7fe54806bef0 2023-07-02T08:59:29.957 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.953+0000 7fe52ffff700 1 -- 172.21.15.164:0/1041874015 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1092+0+0 (crc 0 0 0) 0x7fe538018040 con 0x7fe54806bef0 2023-07-02T08:59:29.957 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.954+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5481a65f0 con 0x7fe54806bef0 2023-07-02T08:59:29.958 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:29.956+0000 7fe52ffff700 1 -- 172.21.15.164:0/1041874015 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (crc 0 0 0) 0x7fe538012750 con 0x7fe54806bef0 2023-07-02T08:59:30.038 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.038+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "647cd0e3-4f38-4dc4-aeb7-d4d2784493da", "id": 3} v 0) v1 -- 0x7fe5481aadf0 con 0x7fe54806bef0 2023-07-02T08:59:30.046 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.047+0000 7fe52ffff700 1 -- 172.21.15.164:0/1041874015 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "647cd0e3-4f38-4dc4-aeb7-d4d2784493da", "id": 3}]=0 v7) v1 ==== 112+0+2 (crc 0 0 0) 0x7fe5380129c0 con 0x7fe54806bef0 2023-07-02T08:59:30.046 INFO:teuthology.orchestra.run.smithi164.stdout:3 2023-07-02T08:59:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.047+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 msgr2=0x7fe5481a5b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.047+0000 7fe54dd84700 1 --2- 172.21.15.164:0/1041874015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe5481a5b30 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.047+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 shutdown_connections 2023-07-02T08:59:30.048 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.047+0000 7fe54dd84700 1 --2- 172.21.15.164:0/1041874015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe54806bef0 0x7fe5481a5b30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:30.048 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.047+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 >> 172.21.15.164:0/1041874015 conn(0x7fe54806f720 msgr2=0x7fe548115ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:30.048 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.047+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 shutdown_connections 2023-07-02T08:59:30.048 INFO:teuthology.orchestra.run.smithi164.stderr:2023-07-02T08:59:30.047+0000 7fe54dd84700 1 -- 172.21.15.164:0/1041874015 wait complete. 2023-07-02T08:59:30.059 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-07-02T08:59:30.097 INFO:tasks.ceph.osd.0:Restarting daemon 2023-07-02T08:59:30.098 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-07-02T08:59:30.140 INFO:tasks.ceph.osd.0:Started 2023-07-02T08:59:30.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-07-02T08:59:30.174 INFO:tasks.ceph.osd.1:Restarting daemon 2023-07-02T08:59:30.175 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-07-02T08:59:30.217 INFO:tasks.ceph.osd.1:Started 2023-07-02T08:59:30.217 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-07-02T08:59:30.249 INFO:tasks.ceph.osd.2:Restarting daemon 2023-07-02T08:59:30.250 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-07-02T08:59:30.291 INFO:tasks.ceph.osd.2:Started 2023-07-02T08:59:30.291 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2023-07-02T08:59:30.323 INFO:tasks.ceph.osd.3:Restarting daemon 2023-07-02T08:59:30.323 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2023-07-02T08:59:30.365 INFO:tasks.ceph.osd.3:Started 2023-07-02T08:59:30.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T08:59:30.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:30.471+0000 7f5b939d01c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-02T08:59:30.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.511+0000 7f2590372700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:30.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.516+0000 7f2590372700 1 Processor -- start 2023-07-02T08:59:30.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.517+0000 7f2590372700 1 -- start start 2023-07-02T08:59:30.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.517+0000 7f2590372700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2588133360 0x7f258809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:30.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.517+0000 7f2590372700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258809e260 con 0x7f2588132f50 2023-07-02T08:59:30.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.517+0000 7f2590372700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25880a1640 con 0x7f2588133360 2023-07-02T08:59:30.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258e10e700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2588132f50 0x7f258809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52472/0 (socket says 172.21.15.103:52472) 2023-07-02T08:59:30.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258e10e700 1 -- 172.21.15.103:0/3987045161 learned_addr learned my addr 172.21.15.103:0/3987045161 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:30.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d90d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2588133360 0x7f258809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:30.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d10c700 1 -- 172.21.15.103:0/3987045161 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337197796 0 0) 0x7f258809e260 con 0x7f2588132f50 2023-07-02T08:59:30.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d10c700 1 -- 172.21.15.103:0/3987045161 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2574003580 con 0x7f2588132f50 2023-07-02T08:59:30.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d90d700 1 -- 172.21.15.103:0/3987045161 >> v1:172.21.15.103:6789/0 conn(0x7f2588132f50 legacy=0x7f258809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:30.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d90d700 1 -- 172.21.15.103:0/3987045161 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2588132a20 con 0x7f2588133360 2023-07-02T08:59:30.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d90d700 1 --2- 172.21.15.103:0/3987045161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588133360 0x7f258809cc90 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed49c923d94555f1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:30.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d10c700 1 -- 172.21.15.103:0/3987045161 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2578009c80 con 0x7f2588133360 2023-07-02T08:59:30.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d10c700 1 -- 172.21.15.103:0/3987045161 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2578009de0 con 0x7f2588133360 2023-07-02T08:59:30.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.518+0000 7f258d10c700 1 -- 172.21.15.103:0/3987045161 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f257800e360 con 0x7f2588133360 2023-07-02T08:59:30.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.519+0000 7f2590372700 1 -- 172.21.15.103:0/3987045161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588133360 msgr2=0x7f258809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:30.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.519+0000 7f2590372700 1 --2- 172.21.15.103:0/3987045161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588133360 0x7f258809cc90 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:30.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.519+0000 7f2590372700 1 -- 172.21.15.103:0/3987045161 shutdown_connections 2023-07-02T08:59:30.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.519+0000 7f2590372700 1 --2- 172.21.15.103:0/3987045161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588133360 0x7f258809cc90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:30.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.519+0000 7f2590372700 1 -- 172.21.15.103:0/3987045161 >> 172.21.15.103:0/3987045161 conn(0x7f25880a2080 msgr2=0x7f25880a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:30.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.519+0000 7f2590372700 1 -- 172.21.15.103:0/3987045161 shutdown_connections 2023-07-02T08:59:30.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.520+0000 7f2590372700 1 -- 172.21.15.103:0/3987045161 wait complete. 2023-07-02T08:59:30.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.520+0000 7f2590372700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:30.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.521+0000 7f2590372700 1 Processor -- start 2023-07-02T08:59:30.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.521+0000 7f2590372700 1 -- start start 2023-07-02T08:59:30.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.521+0000 7f2590372700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588132f50 0x7f258812c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:30.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.521+0000 7f2590372700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258812beb0 con 0x7f2588132f50 2023-07-02T08:59:30.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f258e10e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588132f50 0x7f258812c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:30.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f258e10e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588132f50 0x7f258812c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37116/0 (socket says 172.21.15.103:37116) 2023-07-02T08:59:30.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f258e10e700 1 -- 172.21.15.103:0/3137756743 learned_addr learned my addr 172.21.15.103:0/3137756743 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:30.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f258e10e700 1 -- 172.21.15.103:0/3137756743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258806fb30 con 0x7f2588132f50 2023-07-02T08:59:30.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f258e10e700 1 --2- 172.21.15.103:0/3137756743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588132f50 0x7f258812c270 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:30.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f257effd700 1 -- 172.21.15.103:0/3137756743 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2584005a60 con 0x7f2588132f50 2023-07-02T08:59:30.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f257effd700 1 -- 172.21.15.103:0/3137756743 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2584005bc0 con 0x7f2588132f50 2023-07-02T08:59:30.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f257effd700 1 -- 172.21.15.103:0/3137756743 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2584005e60 con 0x7f2588132f50 2023-07-02T08:59:30.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.522+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258812bb10 con 0x7f2588132f50 2023-07-02T08:59:30.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.523+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258806f5f0 con 0x7f2588132f50 2023-07-02T08:59:30.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.523+0000 7f257effd700 1 -- 172.21.15.103:0/3137756743 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (crc 0 0 0) 0x7f25840073b0 con 0x7f2588132f50 2023-07-02T08:59:30.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.523+0000 7f257effd700 1 -- 172.21.15.103:0/3137756743 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1207+0+0 (crc 0 0 0) 0x7f2584007660 con 0x7f2588132f50 2023-07-02T08:59:30.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.524+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258812bd00 con 0x7f2588132f50 2023-07-02T08:59:30.528 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.528+0000 7f257effd700 1 -- 172.21.15.103:0/3137756743 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (crc 0 0 0) 0x7f258400c770 con 0x7f2588132f50 2023-07-02T08:59:30.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:30.559+0000 7f5b939d01c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-02T08:59:30.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.599+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25880a1920 con 0x7f2588132f50 2023-07-02T08:59:30.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.599+0000 7f257effd700 1 -- 172.21.15.103:0/3137756743 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) v1 ==== 74+0+4151 (crc 0 0 0) 0x7f258400c910 con 0x7f2588132f50 2023-07-02T08:59:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:30.600 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":7,"fsid":"334b59df-ef47-4729-a173-1813560f60f2","created":"2023-07-02T08:59:27.755922+0000","modified":"2023-07-02T08:59:30.040047+0000","last_up_change":"0.000000","last_in_change":"2023-07-02T08:59:30.040047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9838c66d-5b32-4a3c-b090-4d92ba53e4ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fc54ba3d-23a0-488e-9e81-8d60a083b6eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90ee74d5-14e0-4681-a539-dbb849475cec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"647cd0e3-4f38-4dc4-aeb7-d4d2784493da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T08:59:30.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.600+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588132f50 msgr2=0x7f258812c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:30.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.600+0000 7f2590372700 1 --2- 172.21.15.103:0/3137756743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588132f50 0x7f258812c270 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:30.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.600+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 shutdown_connections 2023-07-02T08:59:30.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.600+0000 7f2590372700 1 --2- 172.21.15.103:0/3137756743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2588132f50 0x7f258812c270 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:30.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.600+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 >> 172.21.15.103:0/3137756743 conn(0x7f25880a2080 msgr2=0x7f25881bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:30.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.600+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 shutdown_connections 2023-07-02T08:59:30.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:30.601+0000 7f2590372700 1 -- 172.21.15.103:0/3137756743 wait complete. 2023-07-02T08:59:30.618 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-07-02T08:59:30.618 INFO:tasks.ceph:Waiting for OSDs to come up 2023-07-02T08:59:30.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:30.727+0000 7f5b939d01c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-02T08:59:30.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:30.814+0000 7f5b939d01c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-02T08:59:31.021 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-07-02T08:59:31.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:31.031+0000 7f5b939d01c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-02T08:59:31.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:31.123+0000 7f5b939d01c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-02T08:59:31.144 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.141+0000 7f3a3b4fa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:31.145 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.144+0000 7f3a3b4fa700 1 Processor -- start 2023-07-02T08:59:31.145 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.145+0000 7f3a3b4fa700 1 -- start start 2023-07-02T08:59:31.146 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.145+0000 7f3a3b4fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3a34132c90 0x7f3a34137a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:31.146 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.145+0000 7f3a3b4fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3406d580 con 0x7f3a3409e640 2023-07-02T08:59:31.147 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.145+0000 7f3a3b4fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3409de20 con 0x7f3a34132c90 2023-07-02T08:59:31.147 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a38a95700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3a34132c90 0x7f3a34137a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:31.147 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a38a95700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3a34132c90 0x7f3a34137a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37130/0 (socket says 172.21.15.103:37130) 2023-07-02T08:59:31.147 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a38a95700 1 -- 172.21.15.103:0/1992496956 learned_addr learned my addr 172.21.15.103:0/1992496956 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:31.147 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a39296700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3a3409e640 0x7f3a34132580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52484/0 (socket says 172.21.15.103:52484) 2023-07-02T08:59:31.147 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a38a95700 1 -- 172.21.15.103:0/1992496956 >> v1:172.21.15.103:6789/0 conn(0x7f3a3409e640 legacy=0x7f3a34132580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:31.147 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a38a95700 1 -- 172.21.15.103:0/1992496956 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3406ce40 con 0x7f3a34132c90 2023-07-02T08:59:31.148 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a2bfff700 1 -- 172.21.15.103:0/1992496956 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123212387 0 0) 0x7f3a3406d580 con 0x7f3a3409e640 2023-07-02T08:59:31.148 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a38a95700 1 --2- 172.21.15.103:0/1992496956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a34132c90 0x7f3a34137a40 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a90aee7f7550d07 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:31.148 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a2bfff700 1 -- 172.21.15.103:0/1992496956 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3a24009d90 con 0x7f3a34132c90 2023-07-02T08:59:31.148 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a2bfff700 1 -- 172.21.15.103:0/1992496956 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3a2400e230 con 0x7f3a34132c90 2023-07-02T08:59:31.148 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.146+0000 7f3a2bfff700 1 -- 172.21.15.103:0/1992496956 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3a2400e4d0 con 0x7f3a34132c90 2023-07-02T08:59:31.150 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.147+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/1992496956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a34132c90 msgr2=0x7f3a34137a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:31.150 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.147+0000 7f3a3b4fa700 1 --2- 172.21.15.103:0/1992496956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a34132c90 0x7f3a34137a40 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:31.150 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.147+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/1992496956 shutdown_connections 2023-07-02T08:59:31.150 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.147+0000 7f3a3b4fa700 1 --2- 172.21.15.103:0/1992496956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a34132c90 0x7f3a34137a40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:31.150 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.147+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/1992496956 >> 172.21.15.103:0/1992496956 conn(0x7f3a340a5a20 msgr2=0x7f3a340a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:31.151 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.147+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/1992496956 shutdown_connections 2023-07-02T08:59:31.151 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.147+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/1992496956 wait complete. 2023-07-02T08:59:31.151 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.147+0000 7f3a3b4fa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:31.151 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.148+0000 7f3a3b4fa700 1 Processor -- start 2023-07-02T08:59:31.151 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.148+0000 7f3a3b4fa700 1 -- start start 2023-07-02T08:59:31.151 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a3b4fa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a3409e640 0x7f3a3411e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:31.151 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a3b4fa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a34132240 con 0x7f3a3409e640 2023-07-02T08:59:31.152 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a39296700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a3409e640 0x7f3a3411e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:31.152 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a39296700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a3409e640 0x7f3a3411e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37146/0 (socket says 172.21.15.103:37146) 2023-07-02T08:59:31.152 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a39296700 1 -- 172.21.15.103:0/2312674779 learned_addr learned my addr 172.21.15.103:0/2312674779 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:31.152 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a39296700 1 -- 172.21.15.103:0/2312674779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3411e1a0 con 0x7f3a3409e640 2023-07-02T08:59:31.152 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a39296700 1 --2- 172.21.15.103:0/2312674779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a3409e640 0x7f3a3411e530 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:31.152 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a29ffb700 1 -- 172.21.15.103:0/2312674779 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3a30004b80 con 0x7f3a3409e640 2023-07-02T08:59:31.152 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a29ffb700 1 -- 172.21.15.103:0/2312674779 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3a30004ce0 con 0x7f3a3409e640 2023-07-02T08:59:31.153 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a34121a70 con 0x7f3a3409e640 2023-07-02T08:59:31.153 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a29ffb700 1 -- 172.21.15.103:0/2312674779 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3a3000c550 con 0x7f3a3409e640 2023-07-02T08:59:31.153 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.149+0000 7f3a29ffb700 1 -- 172.21.15.103:0/2312674779 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (crc 0 0 0) 0x7f3a3000c9b0 con 0x7f3a3409e640 2023-07-02T08:59:31.153 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.150+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3411fe60 con 0x7f3a3409e640 2023-07-02T08:59:31.153 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.150+0000 7f3a29ffb700 1 -- 172.21.15.103:0/2312674779 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1207+0+0 (crc 0 0 0) 0x7f3a3000d150 con 0x7f3a3409e640 2023-07-02T08:59:31.154 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.151+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3406b9b0 con 0x7f3a3409e640 2023-07-02T08:59:31.154 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.153+0000 7f3a29ffb700 1 -- 172.21.15.103:0/2312674779 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (crc 0 0 0) 0x7f3a3000d310 con 0x7f3a3409e640 2023-07-02T08:59:31.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:31.209+0000 7f5b939d01c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-02T08:59:31.221 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.221+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a34061ea0 con 0x7f3a3409e640 2023-07-02T08:59:31.222 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.222+0000 7f3a29ffb700 1 -- 172.21.15.103:0/2312674779 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) v1 ==== 74+0+4151 (crc 0 0 0) 0x7f3a3000d4b0 con 0x7f3a3409e640 2023-07-02T08:59:31.222 INFO:teuthology.misc.health.smithi103.stdout: 2023-07-02T08:59:31.222 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":7,"fsid":"334b59df-ef47-4729-a173-1813560f60f2","created":"2023-07-02T08:59:27.755922+0000","modified":"2023-07-02T08:59:30.040047+0000","last_up_change":"0.000000","last_in_change":"2023-07-02T08:59:30.040047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9838c66d-5b32-4a3c-b090-4d92ba53e4ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fc54ba3d-23a0-488e-9e81-8d60a083b6eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90ee74d5-14e0-4681-a539-dbb849475cec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"647cd0e3-4f38-4dc4-aeb7-d4d2784493da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T08:59:31.223 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.223+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a3409e640 msgr2=0x7f3a3411e530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:31.223 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.223+0000 7f3a3b4fa700 1 --2- 172.21.15.103:0/2312674779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a3409e640 0x7f3a3411e530 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:31.223 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.223+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 shutdown_connections 2023-07-02T08:59:31.223 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.223+0000 7f3a3b4fa700 1 --2- 172.21.15.103:0/2312674779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a3409e640 0x7f3a3411e530 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:31.224 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.223+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 >> 172.21.15.103:0/2312674779 conn(0x7f3a340a5a20 msgr2=0x7f3a341ba530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:31.224 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.223+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 shutdown_connections 2023-07-02T08:59:31.224 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:31.223+0000 7f3a3b4fa700 1 -- 172.21.15.103:0/2312674779 wait complete. 2023-07-02T08:59:31.235 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-07-02T08:59:31.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:31.308+0000 7f5b939d01c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-02T08:59:31.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:31.504+0000 7f5b939d01c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-02T08:59:31.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:31.598+0000 7f5b939d01c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-02T08:59:31.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:31.916+0000 7f5b939d01c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-02T08:59:31.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:31.996+0000 7f5b939d01c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-02T08:59:32.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:32.552+0000 7f5b939d01c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-02T08:59:32.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:32.889+0000 7f5b939d01c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-02T08:59:33.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:33.038+0000 7f5b939d01c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-02T08:59:33.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:33.434+0000 7f5b939d01c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-02T08:59:33.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:33.583+0000 7f5b939d01c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-02T08:59:33.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:33.738+0000 7f5b939d01c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-02T08:59:34.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:34.051+0000 7f5b939d01c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-02T08:59:35.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T08:59:35.064+0000 7f5b578ec700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-07-02T08:59:37.638 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-07-02T08:59:37.776 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.773+0000 7f3378ee2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:37.777 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.777+0000 7f3378ee2700 1 Processor -- start 2023-07-02T08:59:37.777 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.777+0000 7f3378ee2700 1 -- start start 2023-07-02T08:59:37.778 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f3378ee2700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f337412f150 0x7f337412f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:37.779 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f3378ee2700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337409a340 con 0x7f3374098a00 2023-07-02T08:59:37.779 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f3378ee2700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337412ed00 con 0x7f337412f150 2023-07-02T08:59:37.779 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f337259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f337412f150 0x7f337412f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:37.779 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f3371d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3374098a00 0x7f3374098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43028/0 (socket says 172.21.15.103:43028) 2023-07-02T08:59:37.779 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f3371d9b700 1 -- 172.21.15.103:0/3509955151 learned_addr learned my addr 172.21.15.103:0/3509955151 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:37.779 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f337259c700 1 -- 172.21.15.103:0/3509955151 >> v1:172.21.15.103:6789/0 conn(0x7f3374098a00 legacy=0x7f3374098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:37.779 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f337259c700 1 -- 172.21.15.103:0/3509955151 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337409a780 con 0x7f337412f150 2023-07-02T08:59:37.780 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f337259c700 1 --2- 172.21.15.103:0/3509955151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f337412f150 0x7f337412f560 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6c7f507c83c00c server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:37.780 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f337159a700 1 -- 172.21.15.103:0/3509955151 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3368009cd0 con 0x7f337412f150 2023-07-02T08:59:37.780 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f337159a700 1 -- 172.21.15.103:0/3509955151 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3368009e30 con 0x7f337412f150 2023-07-02T08:59:37.780 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.778+0000 7f337159a700 1 -- 172.21.15.103:0/3509955151 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f336800e360 con 0x7f337412f150 2023-07-02T08:59:37.781 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.779+0000 7f3378ee2700 1 -- 172.21.15.103:0/3509955151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f337412f150 msgr2=0x7f337412f560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:37.782 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.779+0000 7f3378ee2700 1 --2- 172.21.15.103:0/3509955151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f337412f150 0x7f337412f560 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:37.782 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.779+0000 7f3378ee2700 1 -- 172.21.15.103:0/3509955151 shutdown_connections 2023-07-02T08:59:37.782 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.779+0000 7f3378ee2700 1 --2- 172.21.15.103:0/3509955151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f337412f150 0x7f337412f560 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:37.782 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.779+0000 7f3378ee2700 1 -- 172.21.15.103:0/3509955151 >> 172.21.15.103:0/3509955151 conn(0x7f337406a290 msgr2=0x7f337406a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:37.782 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.779+0000 7f3378ee2700 1 -- 172.21.15.103:0/3509955151 shutdown_connections 2023-07-02T08:59:37.782 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.779+0000 7f3378ee2700 1 -- 172.21.15.103:0/3509955151 wait complete. 2023-07-02T08:59:37.782 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.779+0000 7f3378ee2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:37.783 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.780+0000 7f3378ee2700 1 Processor -- start 2023-07-02T08:59:37.783 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.781+0000 7f3378ee2700 1 -- start start 2023-07-02T08:59:37.783 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.781+0000 7f3378ee2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3374098a00 0x7f3374070960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:37.783 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.781+0000 7f3378ee2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337409ca00 con 0x7f3374098a00 2023-07-02T08:59:37.783 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.781+0000 7f337259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3374098a00 0x7f3374070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:37.783 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.781+0000 7f337259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3374098a00 0x7f3374070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55056/0 (socket says 172.21.15.103:55056) 2023-07-02T08:59:37.784 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.781+0000 7f337259c700 1 -- 172.21.15.103:0/2684914759 learned_addr learned my addr 172.21.15.103:0/2684914759 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:37.785 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.782+0000 7f337259c700 1 -- 172.21.15.103:0/2684914759 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33741176d0 con 0x7f3374098a00 2023-07-02T08:59:37.785 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.782+0000 7f337259c700 1 --2- 172.21.15.103:0/2684914759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3374098a00 0x7f3374070960 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:37.785 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.782+0000 7f3362ffd700 1 -- 172.21.15.103:0/2684914759 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3368013740 con 0x7f3374098a00 2023-07-02T08:59:37.785 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.782+0000 7f3362ffd700 1 -- 172.21.15.103:0/2684914759 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f33680138a0 con 0x7f3374098a00 2023-07-02T08:59:37.785 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.782+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3374071d60 con 0x7f3374098a00 2023-07-02T08:59:37.786 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.782+0000 7f3362ffd700 1 -- 172.21.15.103:0/2684914759 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3368013b40 con 0x7f3374098a00 2023-07-02T08:59:37.786 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.782+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3374118250 con 0x7f3374098a00 2023-07-02T08:59:37.786 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.783+0000 7f3362ffd700 1 -- 172.21.15.103:0/2684914759 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7f3368013cf0 con 0x7f3374098a00 2023-07-02T08:59:37.786 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.783+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3374061ea0 con 0x7f3374098a00 2023-07-02T08:59:37.786 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.783+0000 7f3362ffd700 1 --2- 172.21.15.103:0/2684914759 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3354037730 0x7f3354039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:37.786 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.783+0000 7f3362ffd700 1 -- 172.21.15.103:0/2684914759 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2319+0+0 (crc 0 0 0) 0x7f3368018040 con 0x7f3374098a00 2023-07-02T08:59:37.786 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.783+0000 7f3371d9b700 1 --2- 172.21.15.103:0/2684914759 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3354037730 0x7f3354039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:37.787 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.784+0000 7f3371d9b700 1 --2- 172.21.15.103:0/2684914759 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3354037730 0x7f3354039c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:37.788 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.788+0000 7f3362ffd700 1 -- 172.21.15.103:0/2684914759 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f33680189b0 con 0x7f3374098a00 2023-07-02T08:59:37.981 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.981+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f337406b3a0 con 0x7f3374098a00 2023-07-02T08:59:37.982 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.982+0000 7f3362ffd700 1 -- 172.21.15.103:0/2684914759 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+6929 (crc 0 0 0) 0x7f336800f2a0 con 0x7f3374098a00 2023-07-02T08:59:37.982 INFO:teuthology.misc.health.smithi103.stdout: 2023-07-02T08:59:37.982 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":11,"fsid":"334b59df-ef47-4729-a173-1813560f60f2","created":"2023-07-02T08:59:27.755922+0000","modified":"2023-07-02T08:59:37.739620+0000","last_up_change":"2023-07-02T08:59:34.727323+0000","last_in_change":"2023-07-02T08:59:30.040047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-02T08:59:36.100770+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9838c66d-5b32-4a3c-b090-4d92ba53e4ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3256834854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":3256834854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3256834854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":3256834854}]},"public_addr":"172.21.15.103:6800/3256834854","cluster_addr":"172.21.15.103:6801/3256834854","heartbeat_back_addr":"172.21.15.103:6804/3256834854","heartbeat_front_addr":"172.21.15.103:6805/3256834854","state":["exists","up"]},{"osd":1,"uuid":"fc54ba3d-23a0-488e-9e81-8d60a083b6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1949274160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":1949274160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1949274160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":1949274160}]},"public_addr":"172.21.15.103:6802/1949274160","cluster_addr":"172.21.15.103:6803/1949274160","heartbeat_back_addr":"172.21.15.103:6806/1949274160","heartbeat_front_addr":"172.21.15.103:6807/1949274160","state":["exists","up"]},{"osd":2,"uuid":"90ee74d5-14e0-4681-a539-dbb849475cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":4222444210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":4222444210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":4222444210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":4222444210}]},"public_addr":"172.21.15.164:6800/4222444210","cluster_addr":"172.21.15.164:6801/4222444210","heartbeat_back_addr":"172.21.15.164:6805/4222444210","heartbeat_front_addr":"172.21.15.164:6804/4222444210","state":["exists","up"]},{"osd":3,"uuid":"647cd0e3-4f38-4dc4-aeb7-d4d2784493da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3618079103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":3618079103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3618079103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":3618079103}]},"public_addr":"172.21.15.164:6802/3618079103","cluster_addr":"172.21.15.164:6803/3618079103","heartbeat_back_addr":"172.21.15.164:6806/3618079103","heartbeat_front_addr":"172.21.15.164:6807/3618079103","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T08:59:37.984 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3354037730 msgr2=0x7f3354039c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:37.984 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 --2- 172.21.15.103:0/2684914759 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3354037730 0x7f3354039c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:37.984 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3374098a00 msgr2=0x7f3374070960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:37.985 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 --2- 172.21.15.103:0/2684914759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3374098a00 0x7f3374070960 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:37.985 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 shutdown_connections 2023-07-02T08:59:37.985 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 --2- 172.21.15.103:0/2684914759 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3354037730 0x7f3354039c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:37.985 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 --2- 172.21.15.103:0/2684914759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3374098a00 0x7f3374070960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:37.985 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 >> 172.21.15.103:0/2684914759 conn(0x7f337406a290 msgr2=0x7f33741b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:37.985 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 shutdown_connections 2023-07-02T08:59:37.985 INFO:teuthology.misc.health.smithi103.stderr:2023-07-02T08:59:37.984+0000 7f3378ee2700 1 -- 172.21.15.103:0/2684914759 wait complete. 2023-07-02T08:59:37.995 DEBUG:teuthology.misc:4 of 4 OSDs are up 2023-07-02T08:59:37.995 INFO:tasks.ceph:Creating RBD pool 2023-07-02T08:59:37.995 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd pool create rbd 8 2023-07-02T08:59:38.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.120+0000 7f21f70bf700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:38.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.127+0000 7f21f70bf700 1 Processor -- start 2023-07-02T08:59:38.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.127+0000 7f21f70bf700 1 -- start start 2023-07-02T08:59:38.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.127+0000 7f21f70bf700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f21f0099a10 0x7f21f0133d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:38.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.127+0000 7f21f70bf700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f012ef50 con 0x7f21f006c390 2023-07-02T08:59:38.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.127+0000 7f21f70bf700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f009a570 con 0x7f21f0099a10 2023-07-02T08:59:38.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21effff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f21f0099a10 0x7f21f0133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:38.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21f4e5b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f21f006c390 0x7f21f012f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43042/0 (socket says 172.21.15.103:43042) 2023-07-02T08:59:38.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21f4e5b700 1 -- 172.21.15.103:0/3632545360 learned_addr learned my addr 172.21.15.103:0/3632545360 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:38.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21effff700 1 -- 172.21.15.103:0/3632545360 >> v1:172.21.15.103:6789/0 conn(0x7f21f006c390 legacy=0x7f21f012f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:38.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21effff700 1 -- 172.21.15.103:0/3632545360 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f006d1e0 con 0x7f21f0099a10 2023-07-02T08:59:38.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21effff700 1 --2- 172.21.15.103:0/3632545360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f0099a10 0x7f21f0133d00 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7246826d9d21799d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:38.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21ef7fe700 1 -- 172.21.15.103:0/3632545360 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f21dc009cd0 con 0x7f21f0099a10 2023-07-02T08:59:38.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21ef7fe700 1 -- 172.21.15.103:0/3632545360 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f21dc009e30 con 0x7f21f0099a10 2023-07-02T08:59:38.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.128+0000 7f21ef7fe700 1 -- 172.21.15.103:0/3632545360 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f21dc00e360 con 0x7f21f0099a10 2023-07-02T08:59:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.129+0000 7f21f70bf700 1 -- 172.21.15.103:0/3632545360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f0099a10 msgr2=0x7f21f0133d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.129+0000 7f21f70bf700 1 --2- 172.21.15.103:0/3632545360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f0099a10 0x7f21f0133d00 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.129+0000 7f21f70bf700 1 -- 172.21.15.103:0/3632545360 shutdown_connections 2023-07-02T08:59:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.129+0000 7f21f70bf700 1 --2- 172.21.15.103:0/3632545360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f0099a10 0x7f21f0133d00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.129+0000 7f21f70bf700 1 -- 172.21.15.103:0/3632545360 >> 172.21.15.103:0/3632545360 conn(0x7f21f00a0c00 msgr2=0x7f21f00a1010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.129+0000 7f21f70bf700 1 -- 172.21.15.103:0/3632545360 shutdown_connections 2023-07-02T08:59:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.129+0000 7f21f70bf700 1 -- 172.21.15.103:0/3632545360 wait complete. 2023-07-02T08:59:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.129+0000 7f21f70bf700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.130+0000 7f21f70bf700 1 Processor -- start 2023-07-02T08:59:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.130+0000 7f21f70bf700 1 -- start start 2023-07-02T08:59:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21f70bf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f006c390 0x7f21f0133130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21f70bf700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0134690 con 0x7f21f006c390 2023-07-02T08:59:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21f4e5b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f006c390 0x7f21f0133130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21f4e5b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f006c390 0x7f21f0133130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55060/0 (socket says 172.21.15.103:55060) 2023-07-02T08:59:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21f4e5b700 1 -- 172.21.15.103:0/4003031927 learned_addr learned my addr 172.21.15.103:0/4003031927 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21f4e5b700 1 -- 172.21.15.103:0/4003031927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f0135460 con 0x7f21f006c390 2023-07-02T08:59:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21f4e5b700 1 --2- 172.21.15.103:0/4003031927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f006c390 0x7f21f0133130 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21ed7fa700 1 -- 172.21.15.103:0/4003031927 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f21e4005a60 con 0x7f21f006c390 2023-07-02T08:59:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.131+0000 7f21ed7fa700 1 -- 172.21.15.103:0/4003031927 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f21e4005bc0 con 0x7f21f006c390 2023-07-02T08:59:38.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.132+0000 7f21ed7fa700 1 -- 172.21.15.103:0/4003031927 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f21e4005e60 con 0x7f21f006c390 2023-07-02T08:59:38.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.132+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f0135e50 con 0x7f21f006c390 2023-07-02T08:59:38.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.132+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f01348f0 con 0x7f21f006c390 2023-07-02T08:59:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.132+0000 7f21ed7fa700 1 -- 172.21.15.103:0/4003031927 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7f21e400c7b0 con 0x7f21f006c390 2023-07-02T08:59:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.132+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f006e940 con 0x7f21f006c390 2023-07-02T08:59:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.133+0000 7f21ed7fa700 1 --2- 172.21.15.103:0/4003031927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21d4037730 0x7f21d4039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.133+0000 7f21ed7fa700 1 -- 172.21.15.103:0/4003031927 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2319+0+0 (crc 0 0 0) 0x7f21e40468f0 con 0x7f21f006c390 2023-07-02T08:59:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.133+0000 7f21effff700 1 --2- 172.21.15.103:0/4003031927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21d4037730 0x7f21d4039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.134+0000 7f21effff700 1 --2- 172.21.15.103:0/4003031927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21d4037730 0x7f21d4039c40 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:38.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.137+0000 7f21ed7fa700 1 -- 172.21.15.103:0/4003031927 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f21e400a4a0 con 0x7f21f006c390 2023-07-02T08:59:38.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.328+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f21f0061ea0 con 0x7f21f006c390 2023-07-02T08:59:38.747 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.747+0000 7f21ed7fa700 1 -- 172.21.15.103:0/4003031927 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v12) v1 ==== 109+0+0 (crc 0 0 0) 0x7f21e400a4a0 con 0x7f21f006c390 2023-07-02T08:59:38.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21d4037730 msgr2=0x7f21d4039c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:38.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 --2- 172.21.15.103:0/4003031927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21d4037730 0x7f21d4039c40 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:38.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f006c390 msgr2=0x7f21f0133130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:38.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 --2- 172.21.15.103:0/4003031927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f006c390 0x7f21f0133130 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:38.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 shutdown_connections 2023-07-02T08:59:38.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 --2- 172.21.15.103:0/4003031927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21d4037730 0x7f21d4039c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:38.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 --2- 172.21.15.103:0/4003031927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f006c390 0x7f21f0133130 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:38.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 >> 172.21.15.103:0/4003031927 conn(0x7f21f00a0c00 msgr2=0x7f21f009ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:38.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 shutdown_connections 2023-07-02T08:59:38.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.751+0000 7f21f70bf700 1 -- 172.21.15.103:0/4003031927 wait complete. 2023-07-02T08:59:38.753 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'rbd' created 2023-07-02T08:59:38.764 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-07-02T08:59:38.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.902+0000 7fd90e786700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:38.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.911+0000 7fd90e786700 1 Processor -- start 2023-07-02T08:59:38.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.911+0000 7fd90e786700 1 -- start start 2023-07-02T08:59:38.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.912+0000 7fd90e786700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd90809cbd0 0x7fd90809cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:38.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.912+0000 7fd90e786700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90809c580 con 0x7fd90812f300 2023-07-02T08:59:38.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.912+0000 7fd90e786700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90809d520 con 0x7fd90809cbd0 2023-07-02T08:59:38.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.912+0000 7fd907fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd90809cbd0 0x7fd90809cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:38.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.912+0000 7fd9077fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd90812f300 0x7fd90812f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43056/0 (socket says 172.21.15.103:43056) 2023-07-02T08:59:38.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.912+0000 7fd9077fe700 1 -- 172.21.15.103:0/373832251 learned_addr learned my addr 172.21.15.103:0/373832251 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:38.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd907fff700 1 -- 172.21.15.103:0/373832251 >> v1:172.21.15.103:6789/0 conn(0x7fd90812f300 legacy=0x7fd90812f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:38.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd907fff700 1 -- 172.21.15.103:0/373832251 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080a0d00 con 0x7fd90809cbd0 2023-07-02T08:59:38.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd907fff700 1 --2- 172.21.15.103:0/373832251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd90809cfe0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340e60c767aec8dd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:38.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd906ffd700 1 -- 172.21.15.103:0/373832251 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8f8005d80 con 0x7fd90809cbd0 2023-07-02T08:59:38.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd906ffd700 1 -- 172.21.15.103:0/373832251 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd8f8009930 con 0x7fd90809cbd0 2023-07-02T08:59:38.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd906ffd700 1 -- 172.21.15.103:0/373832251 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8f8009bd0 con 0x7fd90809cbd0 2023-07-02T08:59:38.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd90e786700 1 -- 172.21.15.103:0/373832251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 msgr2=0x7fd90809cfe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:38.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd90e786700 1 --2- 172.21.15.103:0/373832251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd90809cfe0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:38.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd90e786700 1 -- 172.21.15.103:0/373832251 shutdown_connections 2023-07-02T08:59:38.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd90e786700 1 --2- 172.21.15.103:0/373832251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd90809cfe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:38.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.913+0000 7fd90e786700 1 -- 172.21.15.103:0/373832251 >> 172.21.15.103:0/373832251 conn(0x7fd9080a1d40 msgr2=0x7fd9080a2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:38.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.914+0000 7fd90e786700 1 -- 172.21.15.103:0/373832251 shutdown_connections 2023-07-02T08:59:38.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.914+0000 7fd90e786700 1 -- 172.21.15.103:0/373832251 wait complete. 2023-07-02T08:59:38.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.914+0000 7fd90e786700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:38.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.915+0000 7fd90e786700 1 Processor -- start 2023-07-02T08:59:38.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.915+0000 7fd90e786700 1 -- start start 2023-07-02T08:59:38.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.915+0000 7fd90e786700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd9081c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:38.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.915+0000 7fd90e786700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90812d3e0 con 0x7fd90809cbd0 2023-07-02T08:59:38.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd907fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd9081c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:38.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd907fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd9081c4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55078/0 (socket says 172.21.15.103:55078) 2023-07-02T08:59:38.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd907fff700 1 -- 172.21.15.103:0/1949262990 learned_addr learned my addr 172.21.15.103:0/1949262990 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:38.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd907fff700 1 -- 172.21.15.103:0/1949262990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9081c4180 con 0x7fd90809cbd0 2023-07-02T08:59:38.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd907fff700 1 --2- 172.21.15.103:0/1949262990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd9081c4540 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:38.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd904ff9700 1 -- 172.21.15.103:0/1949262990 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8f8014540 con 0x7fd90809cbd0 2023-07-02T08:59:38.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd904ff9700 1 -- 172.21.15.103:0/1949262990 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd8f80146a0 con 0x7fd90809cbd0 2023-07-02T08:59:38.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd904ff9700 1 -- 172.21.15.103:0/1949262990 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8f8014940 con 0x7fd90809cbd0 2023-07-02T08:59:38.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90806e950 con 0x7fd90809cbd0 2023-07-02T08:59:38.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.916+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90812db10 con 0x7fd90809cbd0 2023-07-02T08:59:38.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.917+0000 7fd904ff9700 1 -- 172.21.15.103:0/1949262990 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7fd8f8014b40 con 0x7fd90809cbd0 2023-07-02T08:59:38.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.917+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90806e7a0 con 0x7fd90809cbd0 2023-07-02T08:59:38.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.917+0000 7fd904ff9700 1 --2- 172.21.15.103:0/1949262990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8f40376e0 0x7fd8f4039bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:38.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.917+0000 7fd904ff9700 1 -- 172.21.15.103:0/1949262990 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2661+0+0 (crc 0 0 0) 0x7fd8f8047e20 con 0x7fd90809cbd0 2023-07-02T08:59:38.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.917+0000 7fd9077fe700 1 --2- 172.21.15.103:0/1949262990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8f40376e0 0x7fd8f4039bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:38.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.918+0000 7fd9077fe700 1 --2- 172.21.15.103:0/1949262990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8f40376e0 0x7fd8f4039bf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:38.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:38.922+0000 7fd904ff9700 1 -- 172.21.15.103:0/1949262990 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd8f8006850 con 0x7fd90809cbd0 2023-07-02T08:59:39.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.109+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd90812f0e0 con 0x7fd90809cbd0 2023-07-02T08:59:39.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.753+0000 7fd904ff9700 1 -- 172.21.15.103:0/1949262990 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v13) v1 ==== 173+0+0 (crc 0 0 0) 0x7fd8f8006440 con 0x7fd90809cbd0 2023-07-02T08:59:39.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.755+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8f40376e0 msgr2=0x7fd8f4039bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:39.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.755+0000 7fd90e786700 1 --2- 172.21.15.103:0/1949262990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8f40376e0 0x7fd8f4039bf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:39.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.755+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 msgr2=0x7fd9081c4540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:39.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.755+0000 7fd90e786700 1 --2- 172.21.15.103:0/1949262990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd9081c4540 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:39.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.755+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 shutdown_connections 2023-07-02T08:59:39.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.755+0000 7fd90e786700 1 --2- 172.21.15.103:0/1949262990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8f40376e0 0x7fd8f4039bf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:39.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.755+0000 7fd90e786700 1 --2- 172.21.15.103:0/1949262990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd90809cbd0 0x7fd9081c4540 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:39.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.755+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 >> 172.21.15.103:0/1949262990 conn(0x7fd9080a1d40 msgr2=0x7fd908130600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:39.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.756+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 shutdown_connections 2023-07-02T08:59:39.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.756+0000 7fd90e786700 1 -- 172.21.15.103:0/1949262990 wait complete. 2023-07-02T08:59:39.758 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rbd' on pool 'rbd' 2023-07-02T08:59:39.770 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-07-02T08:59:39.771 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-02T08:59:39.771 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-07-02T08:59:39.771 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-07-02T08:59:39.772 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-07-02T08:59:39.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.910+0000 7fc8ae369700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:39.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.914+0000 7fc8ae369700 1 Processor -- start 2023-07-02T08:59:39.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.914+0000 7fc8ae369700 1 -- start start 2023-07-02T08:59:39.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8ae369700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc8a8128830 0x7fc8a8128c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:39.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8ae369700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a809ad40 con 0x7fc8a8098a00 2023-07-02T08:59:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8ae369700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a806d350 con 0x7fc8a8128830 2023-07-02T08:59:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc8a8128830 0x7fc8a8128c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a77fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc8a8098a00 0x7fc8a8098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43058/0 (socket says 172.21.15.103:43058) 2023-07-02T08:59:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc8a8128830 0x7fc8a8128c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55086/0 (socket says 172.21.15.103:55086) 2023-07-02T08:59:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a77fe700 1 -- 172.21.15.103:0/1105831346 learned_addr learned my addr 172.21.15.103:0/1105831346 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a7fff700 1 -- 172.21.15.103:0/1105831346 >> v1:172.21.15.103:6789/0 conn(0x7fc8a8098a00 legacy=0x7fc8a8098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a7fff700 1 -- 172.21.15.103:0/1105831346 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a809a830 con 0x7fc8a8128830 2023-07-02T08:59:39.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a7fff700 1 --2- 172.21.15.103:0/1105831346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8128830 0x7fc8a8128c40 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3e9ca5d445fe73 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a6ffd700 1 -- 172.21.15.103:0/1105831346 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc898009cd0 con 0x7fc8a8128830 2023-07-02T08:59:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.915+0000 7fc8a6ffd700 1 -- 172.21.15.103:0/1105831346 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc898009e30 con 0x7fc8a8128830 2023-07-02T08:59:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8a6ffd700 1 -- 172.21.15.103:0/1105831346 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc89800e360 con 0x7fc8a8128830 2023-07-02T08:59:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8ae369700 1 -- 172.21.15.103:0/1105831346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8128830 msgr2=0x7fc8a8128c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8ae369700 1 --2- 172.21.15.103:0/1105831346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8128830 0x7fc8a8128c40 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8ae369700 1 -- 172.21.15.103:0/1105831346 shutdown_connections 2023-07-02T08:59:39.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8ae369700 1 --2- 172.21.15.103:0/1105831346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8128830 0x7fc8a8128c40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:39.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8ae369700 1 -- 172.21.15.103:0/1105831346 >> 172.21.15.103:0/1105831346 conn(0x7fc8a809d810 msgr2=0x7fc8a809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:39.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8ae369700 1 -- 172.21.15.103:0/1105831346 shutdown_connections 2023-07-02T08:59:39.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8ae369700 1 -- 172.21.15.103:0/1105831346 wait complete. 2023-07-02T08:59:39.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.916+0000 7fc8ae369700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:39.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.917+0000 7fc8ae369700 1 Processor -- start 2023-07-02T08:59:39.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.917+0000 7fc8ae369700 1 -- start start 2023-07-02T08:59:39.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.918+0000 7fc8ae369700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8098a00 0x7fc8a8119570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:39.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.918+0000 7fc8ae369700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a811b7a0 con 0x7fc8a8098a00 2023-07-02T08:59:39.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.918+0000 7fc8a7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8098a00 0x7fc8a8119570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:39.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.918+0000 7fc8a7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8098a00 0x7fc8a8119570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55098/0 (socket says 172.21.15.103:55098) 2023-07-02T08:59:39.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.918+0000 7fc8a7fff700 1 -- 172.21.15.103:0/2048839720 learned_addr learned my addr 172.21.15.103:0/2048839720 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:39.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.919+0000 7fc8a7fff700 1 -- 172.21.15.103:0/2048839720 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a811aae0 con 0x7fc8a8098a00 2023-07-02T08:59:39.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.919+0000 7fc8a7fff700 1 --2- 172.21.15.103:0/2048839720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8098a00 0x7fc8a8119570 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:39.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.919+0000 7fc8a4ff9700 1 -- 172.21.15.103:0/2048839720 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc898013530 con 0x7fc8a8098a00 2023-07-02T08:59:39.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.919+0000 7fc8a4ff9700 1 -- 172.21.15.103:0/2048839720 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc898013690 con 0x7fc8a8098a00 2023-07-02T08:59:39.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.919+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a8071760 con 0x7fc8a8098a00 2023-07-02T08:59:39.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.919+0000 7fc8a4ff9700 1 -- 172.21.15.103:0/2048839720 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc898013930 con 0x7fc8a8098a00 2023-07-02T08:59:39.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.919+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a811c680 con 0x7fc8a8098a00 2023-07-02T08:59:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.920+0000 7fc8a4ff9700 1 -- 172.21.15.103:0/2048839720 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7fc898013a90 con 0x7fc8a8098a00 2023-07-02T08:59:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.920+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a806dee0 con 0x7fc8a8098a00 2023-07-02T08:59:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.920+0000 7fc8a4ff9700 1 --2- 172.21.15.103:0/2048839720 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc894037730 0x7fc894039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.920+0000 7fc8a4ff9700 1 -- 172.21.15.103:0/2048839720 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc898018040 con 0x7fc8a8098a00 2023-07-02T08:59:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.920+0000 7fc8a77fe700 1 --2- 172.21.15.103:0/2048839720 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc894037730 0x7fc894039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.920+0000 7fc8a77fe700 1 --2- 172.21.15.103:0/2048839720 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc894037730 0x7fc894039c40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:39.925 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:39.925+0000 7fc8a4ff9700 1 -- 172.21.15.103:0/2048839720 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc898018a40 con 0x7fc8a8098a00 2023-07-02T08:59:40.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.176+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc8a809a830 con 0x7fc8a8098a00 2023-07-02T08:59:40.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.178+0000 7fc8a4ff9700 1 -- 172.21.15.103:0/2048839720 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+85987 (crc 0 0 0) 0x7fc8a809a830 con 0x7fc8a8098a00 2023-07-02T08:59:40.179 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:40.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc894037730 msgr2=0x7fc894039c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 --2- 172.21.15.103:0/2048839720 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc894037730 0x7fc894039c40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8098a00 msgr2=0x7fc8a8119570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 --2- 172.21.15.103:0/2048839720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8098a00 0x7fc8a8119570 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 shutdown_connections 2023-07-02T08:59:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 --2- 172.21.15.103:0/2048839720 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc894037730 0x7fc894039c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 --2- 172.21.15.103:0/2048839720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8a8098a00 0x7fc8a8119570 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 >> 172.21.15.103:0/2048839720 conn(0x7fc8a809d810 msgr2=0x7fc8a812e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:40.182 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 shutdown_connections 2023-07-02T08:59:40.182 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.180+0000 7fc8ae369700 1 -- 172.21.15.103:0/2048839720 wait complete. 2023-07-02T08:59:40.192 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":47615948},{"type":"v1","addr":"172.21.15.164:6809","nonce":47615948}]},"active_addr":"172.21.15.164:6809/47615948","active_change":"2023-07-02T08:59:34.053907+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":354102999}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":3631195495}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2848743815}]}]} 2023-07-02T08:59:40.195 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-07-02T08:59:40.195 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-07-02T08:59:40.195 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T08:59:40.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.320+0000 7fa7e5d3f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:40.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.324+0000 7fa7e5d3f700 1 Processor -- start 2023-07-02T08:59:40.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.324+0000 7fa7e5d3f700 1 -- start start 2023-07-02T08:59:40.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7e5d3f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa7e0132f50 0x7fa7e0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7e5d3f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e009e260 con 0x7fa7e009c580 2023-07-02T08:59:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7e5d3f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e00a1640 con 0x7fa7e0132f50 2023-07-02T08:59:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7df7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa7e0132f50 0x7fa7e0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7deffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa7e009c580 0x7fa7e009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43070/0 (socket says 172.21.15.103:43070) 2023-07-02T08:59:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7df7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa7e0132f50 0x7fa7e0133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55104/0 (socket says 172.21.15.103:55104) 2023-07-02T08:59:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7deffd700 1 -- 172.21.15.103:0/1218514288 learned_addr learned my addr 172.21.15.103:0/1218514288 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7df7fe700 1 -- 172.21.15.103:0/1218514288 >> v1:172.21.15.103:6789/0 conn(0x7fa7e009c580 legacy=0x7fa7e009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7df7fe700 1 -- 172.21.15.103:0/1218514288 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e0132a20 con 0x7fa7e0132f50 2023-07-02T08:59:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7de7fc700 1 -- 172.21.15.103:0/1218514288 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172170685 0 0) 0x7fa7e009e260 con 0x7fa7e009c580 2023-07-02T08:59:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7df7fe700 1 --2- 172.21.15.103:0/1218514288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e0132f50 0x7fa7e0133360 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa4ce6f094bb57 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.325+0000 7fa7de7fc700 1 -- 172.21.15.103:0/1218514288 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa7c8009cd0 con 0x7fa7e0132f50 2023-07-02T08:59:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7de7fc700 1 -- 172.21.15.103:0/1218514288 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa7c8009e30 con 0x7fa7e0132f50 2023-07-02T08:59:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7de7fc700 1 -- 172.21.15.103:0/1218514288 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa7c800e360 con 0x7fa7e0132f50 2023-07-02T08:59:40.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/1218514288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e0132f50 msgr2=0x7fa7e0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7e5d3f700 1 --2- 172.21.15.103:0/1218514288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e0132f50 0x7fa7e0133360 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/1218514288 shutdown_connections 2023-07-02T08:59:40.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7e5d3f700 1 --2- 172.21.15.103:0/1218514288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e0132f50 0x7fa7e0133360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/1218514288 >> 172.21.15.103:0/1218514288 conn(0x7fa7e00a2080 msgr2=0x7fa7e00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/1218514288 shutdown_connections 2023-07-02T08:59:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/1218514288 wait complete. 2023-07-02T08:59:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.326+0000 7fa7e5d3f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.327+0000 7fa7e5d3f700 1 Processor -- start 2023-07-02T08:59:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.328+0000 7fa7e5d3f700 1 -- start start 2023-07-02T08:59:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.328+0000 7fa7e5d3f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e009c580 0x7fa7e01ba930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.328+0000 7fa7e5d3f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e0133a00 con 0x7fa7e009c580 2023-07-02T08:59:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.328+0000 7fa7df7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e009c580 0x7fa7e01ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.328+0000 7fa7df7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e009c580 0x7fa7e01ba930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55112/0 (socket says 172.21.15.103:55112) 2023-07-02T08:59:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.328+0000 7fa7df7fe700 1 -- 172.21.15.103:0/928247882 learned_addr learned my addr 172.21.15.103:0/928247882 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.329+0000 7fa7df7fe700 1 -- 172.21.15.103:0/928247882 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e006d210 con 0x7fa7e009c580 2023-07-02T08:59:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.329+0000 7fa7df7fe700 1 --2- 172.21.15.103:0/928247882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e009c580 0x7fa7e01ba930 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.329+0000 7fa7d7fff700 1 -- 172.21.15.103:0/928247882 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa7c8014540 con 0x7fa7e009c580 2023-07-02T08:59:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.329+0000 7fa7d7fff700 1 -- 172.21.15.103:0/928247882 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa7c80146a0 con 0x7fa7e009c580 2023-07-02T08:59:40.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.329+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e006db60 con 0x7fa7e009c580 2023-07-02T08:59:40.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.329+0000 7fa7d7fff700 1 -- 172.21.15.103:0/928247882 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa7c8014940 con 0x7fa7e009c580 2023-07-02T08:59:40.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.329+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e006f770 con 0x7fa7e009c580 2023-07-02T08:59:40.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.330+0000 7fa7d7fff700 1 -- 172.21.15.103:0/928247882 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7fa7c8014af0 con 0x7fa7e009c580 2023-07-02T08:59:40.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.330+0000 7fa7d7fff700 1 --2- 172.21.15.103:0/928247882 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7c0037730 0x7fa7c0039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:40.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.330+0000 7fa7d7fff700 1 -- 172.21.15.103:0/928247882 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa7c8048280 con 0x7fa7e009c580 2023-07-02T08:59:40.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.330+0000 7fa7deffd700 1 --2- 172.21.15.103:0/928247882 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7c0037730 0x7fa7c0039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:40.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.330+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e0061ea0 con 0x7fa7e009c580 2023-07-02T08:59:40.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.330+0000 7fa7deffd700 1 --2- 172.21.15.103:0/928247882 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7c0037730 0x7fa7c0039c40 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:40.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.335+0000 7fa7d7fff700 1 -- 172.21.15.103:0/928247882 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa7c8014df0 con 0x7fa7e009c580 2023-07-02T08:59:40.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.521+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7e00725b0 con 0x7fa7e009c580 2023-07-02T08:59:40.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.522+0000 7fa7d7fff700 1 -- 172.21.15.103:0/928247882 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+8688 (crc 0 0 0) 0x7fa7c8011320 con 0x7fa7e009c580 2023-07-02T08:59:40.522 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:40.522 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":13,"fsid":"334b59df-ef47-4729-a173-1813560f60f2","created":"2023-07-02T08:59:27.755922+0000","modified":"2023-07-02T08:59:39.749248+0000","last_up_change":"2023-07-02T08:59:34.727323+0000","last_in_change":"2023-07-02T08:59:30.040047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-02T08:59:36.100770+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T08:59:38.345612+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9838c66d-5b32-4a3c-b090-4d92ba53e4ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3256834854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":3256834854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3256834854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":3256834854}]},"public_addr":"172.21.15.103:6800/3256834854","cluster_addr":"172.21.15.103:6801/3256834854","heartbeat_back_addr":"172.21.15.103:6804/3256834854","heartbeat_front_addr":"172.21.15.103:6805/3256834854","state":["exists","up"]},{"osd":1,"uuid":"fc54ba3d-23a0-488e-9e81-8d60a083b6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1949274160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":1949274160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1949274160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":1949274160}]},"public_addr":"172.21.15.103:6802/1949274160","cluster_addr":"172.21.15.103:6803/1949274160","heartbeat_back_addr":"172.21.15.103:6806/1949274160","heartbeat_front_addr":"172.21.15.103:6807/1949274160","state":["exists","up"]},{"osd":2,"uuid":"90ee74d5-14e0-4681-a539-dbb849475cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":4222444210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":4222444210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":4222444210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":4222444210}]},"public_addr":"172.21.15.164:6800/4222444210","cluster_addr":"172.21.15.164:6801/4222444210","heartbeat_back_addr":"172.21.15.164:6805/4222444210","heartbeat_front_addr":"172.21.15.164:6804/4222444210","state":["exists","up"]},{"osd":3,"uuid":"647cd0e3-4f38-4dc4-aeb7-d4d2784493da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3618079103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":3618079103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3618079103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":3618079103}]},"public_addr":"172.21.15.164:6802/3618079103","cluster_addr":"172.21.15.164:6803/3618079103","heartbeat_back_addr":"172.21.15.164:6806/3618079103","heartbeat_front_addr":"172.21.15.164:6807/3618079103","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T08:59:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7c0037730 msgr2=0x7fa7c0039c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 --2- 172.21.15.103:0/928247882 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7c0037730 0x7fa7c0039c40 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e009c580 msgr2=0x7fa7e01ba930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 --2- 172.21.15.103:0/928247882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e009c580 0x7fa7e01ba930 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 shutdown_connections 2023-07-02T08:59:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 --2- 172.21.15.103:0/928247882 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7c0037730 0x7fa7c0039c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 --2- 172.21.15.103:0/928247882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7e009c580 0x7fa7e01ba930 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 >> 172.21.15.103:0/928247882 conn(0x7fa7e00a2080 msgr2=0x7fa7e0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:40.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 shutdown_connections 2023-07-02T08:59:40.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.524+0000 7fa7e5d3f700 1 -- 172.21.15.103:0/928247882 wait complete. 2023-07-02T08:59:40.536 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-07-02T08:59:40.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T08:59:40.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.660+0000 7efdcc27f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:40.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.664+0000 7efdcc27f700 1 Processor -- start 2023-07-02T08:59:40.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.664+0000 7efdcc27f700 1 -- start start 2023-07-02T08:59:40.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.665+0000 7efdcc27f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdc4132030 0x7efdc4134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:40.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.665+0000 7efdcc27f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc406df60 con 0x7efdc412f250 2023-07-02T08:59:40.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.665+0000 7efdcc27f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc409a710 con 0x7efdc4132030 2023-07-02T08:59:40.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.665+0000 7efdc981a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdc4132030 0x7efdc4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.665+0000 7efdca01b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7efdc412f250 0x7efdc412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43074/0 (socket says 172.21.15.103:43074) 2023-07-02T08:59:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.665+0000 7efdc981a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdc4132030 0x7efdc4134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55120/0 (socket says 172.21.15.103:55120) 2023-07-02T08:59:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.665+0000 7efdca01b700 1 -- 172.21.15.103:0/2667428913 learned_addr learned my addr 172.21.15.103:0/2667428913 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.666+0000 7efdc981a700 1 -- 172.21.15.103:0/2667428913 >> v1:172.21.15.103:6789/0 conn(0x7efdc412f250 legacy=0x7efdc412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.666+0000 7efdc981a700 1 -- 172.21.15.103:0/2667428913 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc412ed20 con 0x7efdc4132030 2023-07-02T08:59:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.666+0000 7efdc981a700 1 --2- 172.21.15.103:0/2667428913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4132030 0x7efdc4134450 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614da5b05c3ca0ed server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.666+0000 7efdc9019700 1 -- 172.21.15.103:0/2667428913 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdc000ac60 con 0x7efdc4132030 2023-07-02T08:59:40.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.666+0000 7efdc9019700 1 -- 172.21.15.103:0/2667428913 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efdc000adc0 con 0x7efdc4132030 2023-07-02T08:59:40.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.666+0000 7efdc9019700 1 -- 172.21.15.103:0/2667428913 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdc000f470 con 0x7efdc4132030 2023-07-02T08:59:40.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.666+0000 7efdcc27f700 1 -- 172.21.15.103:0/2667428913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4132030 msgr2=0x7efdc4134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.666+0000 7efdcc27f700 1 --2- 172.21.15.103:0/2667428913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4132030 0x7efdc4134450 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.667+0000 7efdcc27f700 1 -- 172.21.15.103:0/2667428913 shutdown_connections 2023-07-02T08:59:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.667+0000 7efdcc27f700 1 --2- 172.21.15.103:0/2667428913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4132030 0x7efdc4134450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.667+0000 7efdcc27f700 1 -- 172.21.15.103:0/2667428913 >> 172.21.15.103:0/2667428913 conn(0x7efdc409d810 msgr2=0x7efdc409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.667+0000 7efdcc27f700 1 -- 172.21.15.103:0/2667428913 shutdown_connections 2023-07-02T08:59:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.667+0000 7efdcc27f700 1 -- 172.21.15.103:0/2667428913 wait complete. 2023-07-02T08:59:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.667+0000 7efdcc27f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.668+0000 7efdcc27f700 1 Processor -- start 2023-07-02T08:59:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.668+0000 7efdcc27f700 1 -- start start 2023-07-02T08:59:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.668+0000 7efdcc27f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc4117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.668+0000 7efdcc27f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4128ef0 con 0x7efdc412f250 2023-07-02T08:59:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.668+0000 7efdca01b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc4117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdca01b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc4117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55134/0 (socket says 172.21.15.103:55134) 2023-07-02T08:59:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdca01b700 1 -- 172.21.15.103:0/3663779608 learned_addr learned my addr 172.21.15.103:0/3663779608 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdca01b700 1 -- 172.21.15.103:0/3663779608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc411b6c0 con 0x7efdc412f250 2023-07-02T08:59:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdca01b700 1 --2- 172.21.15.103:0/3663779608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc4117360 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdb6ffd700 1 -- 172.21.15.103:0/3663779608 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdb8005a60 con 0x7efdc412f250 2023-07-02T08:59:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdb6ffd700 1 -- 172.21.15.103:0/3663779608 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efdb8005bc0 con 0x7efdc412f250 2023-07-02T08:59:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc4118f30 con 0x7efdc412f250 2023-07-02T08:59:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdb6ffd700 1 -- 172.21.15.103:0/3663779608 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdb8005e60 con 0x7efdc412f250 2023-07-02T08:59:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.669+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc4119810 con 0x7efdc412f250 2023-07-02T08:59:40.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.670+0000 7efdb6ffd700 1 -- 172.21.15.103:0/3663779608 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7efdb80070b0 con 0x7efdc412f250 2023-07-02T08:59:40.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.670+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc4128cf0 con 0x7efdc412f250 2023-07-02T08:59:40.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.670+0000 7efdb6ffd700 1 --2- 172.21.15.103:0/3663779608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda8037730 0x7efda8039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:40.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.670+0000 7efdb6ffd700 1 -- 172.21.15.103:0/3663779608 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2672+0+0 (crc 0 0 0) 0x7efdb8046a40 con 0x7efdc412f250 2023-07-02T08:59:40.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.670+0000 7efdc981a700 1 --2- 172.21.15.103:0/3663779608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda8037730 0x7efda8039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:40.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.671+0000 7efdc981a700 1 --2- 172.21.15.103:0/3663779608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda8037730 0x7efda8039c40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:40.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.675+0000 7efdb6ffd700 1 -- 172.21.15.103:0/3663779608 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7efdb800fdf0 con 0x7efdc412f250 2023-07-02T08:59:40.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.867+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdc4061ea0 con 0x7efdc412f250 2023-07-02T08:59:40.868 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.868+0000 7efdb6ffd700 1 -- 172.21.15.103:0/3663779608 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8679 (crc 0 0 0) 0x7efdb800ddd0 con 0x7efdc412f250 2023-07-02T08:59:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-07-02T08:59:40.868 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":14,"fsid":"334b59df-ef47-4729-a173-1813560f60f2","created":"2023-07-02T08:59:27.755922+0000","modified":"2023-07-02T08:59:40.752930+0000","last_up_change":"2023-07-02T08:59:34.727323+0000","last_in_change":"2023-07-02T08:59:30.040047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-02T08:59:36.100770+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T08:59:38.345612+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9838c66d-5b32-4a3c-b090-4d92ba53e4ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3256834854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":3256834854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3256834854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":3256834854}]},"public_addr":"172.21.15.103:6800/3256834854","cluster_addr":"172.21.15.103:6801/3256834854","heartbeat_back_addr":"172.21.15.103:6804/3256834854","heartbeat_front_addr":"172.21.15.103:6805/3256834854","state":["exists","up"]},{"osd":1,"uuid":"fc54ba3d-23a0-488e-9e81-8d60a083b6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1949274160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":1949274160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1949274160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":1949274160}]},"public_addr":"172.21.15.103:6802/1949274160","cluster_addr":"172.21.15.103:6803/1949274160","heartbeat_back_addr":"172.21.15.103:6806/1949274160","heartbeat_front_addr":"172.21.15.103:6807/1949274160","state":["exists","up"]},{"osd":2,"uuid":"90ee74d5-14e0-4681-a539-dbb849475cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":4222444210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":4222444210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":4222444210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":4222444210}]},"public_addr":"172.21.15.164:6800/4222444210","cluster_addr":"172.21.15.164:6801/4222444210","heartbeat_back_addr":"172.21.15.164:6805/4222444210","heartbeat_front_addr":"172.21.15.164:6804/4222444210","state":["exists","up"]},{"osd":3,"uuid":"647cd0e3-4f38-4dc4-aeb7-d4d2784493da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3618079103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":3618079103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3618079103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":3618079103}]},"public_addr":"172.21.15.164:6802/3618079103","cluster_addr":"172.21.15.164:6803/3618079103","heartbeat_back_addr":"172.21.15.164:6806/3618079103","heartbeat_front_addr":"172.21.15.164:6807/3618079103","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T08:59:40.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda8037730 msgr2=0x7efda8039c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 --2- 172.21.15.103:0/3663779608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda8037730 0x7efda8039c40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 msgr2=0x7efdc4117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:40.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 --2- 172.21.15.103:0/3663779608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc4117360 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 shutdown_connections 2023-07-02T08:59:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 --2- 172.21.15.103:0/3663779608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda8037730 0x7efda8039c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 --2- 172.21.15.103:0/3663779608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc4117360 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 >> 172.21.15.103:0/3663779608 conn(0x7efdc409d810 msgr2=0x7efdc412e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 shutdown_connections 2023-07-02T08:59:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:40.870+0000 7efdcc27f700 1 -- 172.21.15.103:0/3663779608 wait complete. 2023-07-02T08:59:40.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-07-02T08:59:40.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-07-02T08:59:40.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-07-02T08:59:40.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-07-02T08:59:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.021+0000 7faf75d40700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.026+0000 7faf75d40700 1 Processor -- start 2023-07-02T08:59:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.026+0000 7faf75d40700 1 -- start start 2023-07-02T08:59:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf75d40700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf7009d750 0x7faf70137c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf75d40700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7009ce10 con 0x7faf7009eb40 2023-07-02T08:59:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf75d40700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7006d3d0 con 0x7faf7009d750 2023-07-02T08:59:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf6effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf7009d750 0x7faf70137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf6effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf7009d750 0x7faf70137c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55146/0 (socket says 172.21.15.103:55146) 2023-07-02T08:59:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf6f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7faf7009eb40 0x7faf7009d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43078/0 (socket says 172.21.15.103:43078) 2023-07-02T08:59:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf6effd700 1 -- 172.21.15.103:0/2063132582 learned_addr learned my addr 172.21.15.103:0/2063132582 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf6effd700 1 -- 172.21.15.103:0/2063132582 >> v1:172.21.15.103:6789/0 conn(0x7faf7009eb40 legacy=0x7faf7009d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf6e7fc700 1 -- 172.21.15.103:0/2063132582 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696757034 0 0) 0x7faf7009ce10 con 0x7faf7009eb40 2023-07-02T08:59:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf6effd700 1 -- 172.21.15.103:0/2063132582 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7009f2f0 con 0x7faf7009d750 2023-07-02T08:59:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.020+0000 7f974097a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.027+0000 7faf6effd700 1 --2- 172.21.15.103:0/2063132582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf70137c40 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aacc79fa019f7dc server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.028+0000 7faf6e7fc700 1 -- 172.21.15.103:0/2063132582 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf64009d90 con 0x7faf7009d750 2023-07-02T08:59:41.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.028+0000 7faf6e7fc700 1 -- 172.21.15.103:0/2063132582 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7faf6400e230 con 0x7faf7009d750 2023-07-02T08:59:41.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.028+0000 7faf6e7fc700 1 -- 172.21.15.103:0/2063132582 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf6400e4d0 con 0x7faf7009d750 2023-07-02T08:59:41.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.028+0000 7f974097a700 1 Processor -- start 2023-07-02T08:59:41.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.028+0000 7f974097a700 1 -- start start 2023-07-02T08:59:41.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.028+0000 7faf75d40700 1 -- 172.21.15.103:0/2063132582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 msgr2=0x7faf70137c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7faf75d40700 1 --2- 172.21.15.103:0/2063132582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf70137c40 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7faf75d40700 1 -- 172.21.15.103:0/2063132582 shutdown_connections 2023-07-02T08:59:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7faf75d40700 1 --2- 172.21.15.103:0/2063132582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf70137c40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7faf75d40700 1 -- 172.21.15.103:0/2063132582 >> 172.21.15.103:0/2063132582 conn(0x7faf700a1a80 msgr2=0x7faf700a1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7faf75d40700 1 -- 172.21.15.103:0/2063132582 shutdown_connections 2023-07-02T08:59:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7faf75d40700 1 -- 172.21.15.103:0/2063132582 wait complete. 2023-07-02T08:59:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7faf75d40700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7f974097a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f973c133360 0x7f973c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7f974097a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c09e260 con 0x7f973c132f50 2023-07-02T08:59:41.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7f974097a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973c0a1640 con 0x7f973c133360 2023-07-02T08:59:41.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7f9739d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f973c133360 0x7f973c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7f973a59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f973c132f50 0x7f973c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43086/0 (socket says 172.21.15.103:43086) 2023-07-02T08:59:41.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7f973a59c700 1 -- 172.21.15.103:0/1470489721 learned_addr learned my addr 172.21.15.103:0/1470489721 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:41.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.029+0000 7f9739d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f973c133360 0x7f973c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55156/0 (socket says 172.21.15.103:55156) 2023-07-02T08:59:41.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7f9739d9b700 1 -- 172.21.15.103:0/1470489721 >> v1:172.21.15.103:6789/0 conn(0x7f973c132f50 legacy=0x7f973c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7f9739d9b700 1 -- 172.21.15.103:0/1470489721 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c132a20 con 0x7f973c133360 2023-07-02T08:59:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7f9739d9b700 1 --2- 172.21.15.103:0/1470489721 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c133360 0x7f973c09cc90 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878ee99c50dcb1af server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7f973959a700 1 -- 172.21.15.103:0/1470489721 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9730009cd0 con 0x7f973c133360 2023-07-02T08:59:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7f973959a700 1 -- 172.21.15.103:0/1470489721 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9730009e30 con 0x7f973c133360 2023-07-02T08:59:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7f973959a700 1 -- 172.21.15.103:0/1470489721 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f973000e360 con 0x7f973c133360 2023-07-02T08:59:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7faf75d40700 1 Processor -- start 2023-07-02T08:59:41.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7faf75d40700 1 -- start start 2023-07-02T08:59:41.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7f974097a700 1 -- 172.21.15.103:0/1470489721 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c133360 msgr2=0x7f973c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7f974097a700 1 --2- 172.21.15.103:0/1470489721 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c133360 0x7f973c09cc90 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7f974097a700 1 -- 172.21.15.103:0/1470489721 shutdown_connections 2023-07-02T08:59:41.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7f974097a700 1 --2- 172.21.15.103:0/1470489721 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c133360 0x7f973c09cc90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7f974097a700 1 -- 172.21.15.103:0/1470489721 >> 172.21.15.103:0/1470489721 conn(0x7f973c0a2080 msgr2=0x7f973c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7f974097a700 1 -- 172.21.15.103:0/1470489721 shutdown_connections 2023-07-02T08:59:41.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7f974097a700 1 -- 172.21.15.103:0/1470489721 wait complete. 2023-07-02T08:59:41.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7faf75d40700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf7011c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.031+0000 7faf75d40700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7011c180 con 0x7faf7009d750 2023-07-02T08:59:41.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.032+0000 7faf6f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf7011c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.032+0000 7faf6f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf7011c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55166/0 (socket says 172.21.15.103:55166) 2023-07-02T08:59:41.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.032+0000 7faf6f7fe700 1 -- 172.21.15.103:0/3955600816 learned_addr learned my addr 172.21.15.103:0/3955600816 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.032+0000 7f974097a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.032+0000 7faf6f7fe700 1 -- 172.21.15.103:0/3955600816 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf701302d0 con 0x7faf7009d750 2023-07-02T08:59:41.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.032+0000 7faf6f7fe700 1 --2- 172.21.15.103:0/3955600816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf7011c5d0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.033+0000 7f974097a700 1 Processor -- start 2023-07-02T08:59:41.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.033+0000 7f974097a700 1 -- start start 2023-07-02T08:59:41.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.028+0000 7fe9cf492700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f974097a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c132f50 0x7f973c12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f974097a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c06d260 con 0x7f973c132f50 2023-07-02T08:59:41.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f973a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c132f50 0x7f973c12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f973a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c132f50 0x7f973c12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55182/0 (socket says 172.21.15.103:55182) 2023-07-02T08:59:41.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f973a59c700 1 -- 172.21.15.103:0/234142246 learned_addr learned my addr 172.21.15.103:0/234142246 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7fe9cf492700 1 Processor -- start 2023-07-02T08:59:41.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f973a59c700 1 -- 172.21.15.103:0/234142246 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c12bb90 con 0x7f973c132f50 2023-07-02T08:59:41.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f973a59c700 1 --2- 172.21.15.103:0/234142246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c132f50 0x7f973c12bea0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f972affd700 1 -- 172.21.15.103:0/234142246 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f972c005a60 con 0x7f973c132f50 2023-07-02T08:59:41.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f972affd700 1 -- 172.21.15.103:0/234142246 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f972c005bc0 con 0x7f973c132f50 2023-07-02T08:59:41.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f972affd700 1 -- 172.21.15.103:0/234142246 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f972c005e60 con 0x7f973c132f50 2023-07-02T08:59:41.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c06f730 con 0x7f973c132f50 2023-07-02T08:59:41.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c06f490 con 0x7f973c132f50 2023-07-02T08:59:41.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf600059a0 con 0x7faf7009d750 2023-07-02T08:59:41.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7faf60005b00 con 0x7faf7009d750 2023-07-02T08:59:41.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7011fd30 con 0x7faf7009d750 2023-07-02T08:59:41.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf60005da0 con 0x7faf7009d750 2023-07-02T08:59:41.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7011bde0 con 0x7faf7009d750 2023-07-02T08:59:41.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.034+0000 7fe9cf492700 1 -- start start 2023-07-02T08:59:41.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7fe9cf492700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe9c809e3c0 0x7fe9c811b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7fe9cf492700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c809f2e0 con 0x7fe9c809d8a0 2023-07-02T08:59:41.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7fe9cf492700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c8099050 con 0x7fe9c809e3c0 2023-07-02T08:59:41.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7fe9cd22e700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe9c809d8a0 0x7fe9c809dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43088/0 (socket says 172.21.15.103:43088) 2023-07-02T08:59:41.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7fe9cd22e700 1 -- 172.21.15.103:0/3258663222 learned_addr learned my addr 172.21.15.103:0/3258663222 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:41.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.035+0000 7fe9cca2d700 1 --2- 172.21.15.103:0/3258663222 >> v2:172.21.15.103:3300/0 conn(0x7fe9c809e3c0 0x7fe9c811b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.036+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068788291 0 0) 0x7fe9c809f2e0 con 0x7fe9c809d8a0 2023-07-02T08:59:41.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.036+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b4003580 con 0x7fe9c809d8a0 2023-07-02T08:59:41.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.036+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faf50000f10 con 0x7faf7009d750 2023-07-02T08:59:41.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.030+0000 7fe8bc4c8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.036+0000 7f972affd700 1 -- 172.21.15.103:0/234142246 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7f972c0070b0 con 0x7f973c132f50 2023-07-02T08:59:41.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.036+0000 7f972affd700 1 --2- 172.21.15.103:0/234142246 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f971c037730 0x7f971c039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.036+0000 7f972affd700 1 -- 172.21.15.103:0/234142246 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f972c046a60 con 0x7f973c132f50 2023-07-02T08:59:41.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7f974097a700 1 --2- 172.21.15.103:0/234142246 >> v2:172.21.15.103:6800/3256834854 conn(0x7f973c1bb260 0x7f973c12c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 --> v2:172.21.15.103:6800/3256834854 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f973c09db00 con 0x7f973c1bb260 2023-07-02T08:59:41.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 <== mon.0 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43409723 0 0) 0x7fe9b4003580 con 0x7fe9c809d8a0 2023-07-02T08:59:41.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 >> v2:172.21.15.103:3300/0 conn(0x7fe9c809e3c0 msgr2=0x7fe9c811b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-02T08:59:41.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9bffff700 1 --2- 172.21.15.103:0/3258663222 >> v2:172.21.15.103:3300/0 conn(0x7fe9c809e3c0 0x7fe9c811b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7faf600070b0 con 0x7faf7009d750 2023-07-02T08:59:41.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7faf57fff700 1 --2- 172.21.15.103:0/3955600816 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf5c037780 0x7faf5c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c809f020 con 0x7fe9c809d8a0 2023-07-02T08:59:41.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 <== mon.0 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (unknown 973785403 0 0) 0x7fe9c40015a0 con 0x7fe9c809d8a0 2023-07-02T08:59:41.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 >> v1:172.21.15.103:6789/0 conn(0x7fe9c809d8a0 legacy=0x7fe9c809dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9bffff700 1 --2- 172.21.15.103:0/3258663222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c8099050 con 0x7fe9b40039d0 2023-07-02T08:59:41.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe9cd22e700 1 --2- 172.21.15.103:0/3258663222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7faf6000d4d0 con 0x7faf7009d750 2023-07-02T08:59:41.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7faf57fff700 1 --2- 172.21.15.103:0/3955600816 >> v2:172.21.15.164:6802/3618079103 conn(0x7faf5c03c770 0x7faf5c03ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 --> v2:172.21.15.164:6802/3618079103 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf5c03f290 con 0x7faf5c03c770 2023-07-02T08:59:41.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (crc 0 0 0) 0x7faf6000d6e0 con 0x7faf7009d750 2023-07-02T08:59:41.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7f9739d9b700 1 --2- 172.21.15.103:0/234142246 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f971c037730 0x7f971c039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7faf6effd700 1 --2- 172.21.15.103:0/3955600816 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf5c037780 0x7faf5c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7f973ad9d700 1 --2- 172.21.15.103:0/234142246 >> v2:172.21.15.103:6800/3256834854 conn(0x7f973c1bb260 0x7f973c12c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-07-02T08:59:41.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7fe9cd22e700 1 -- 172.21.15.103:0/3258663222 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c809f020 con 0x7fe9b40039d0 2023-07-02T08:59:41.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7faf6ffff700 1 --2- 172.21.15.103:0/3955600816 >> v2:172.21.15.164:6802/3618079103 conn(0x7faf5c03c770 0x7faf5c03ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-07-02T08:59:41.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7fe9cd22e700 1 --2- 172.21.15.103:0/3258663222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9b4003e20 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c7b79e12b147b7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7faf6effd700 1 --2- 172.21.15.103:0/3955600816 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf5c037780 0x7faf5c039c90 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe9c4007ca0 con 0x7fe9b40039d0 2023-07-02T08:59:41.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe9c4005930 con 0x7fe9b40039d0 2023-07-02T08:59:41.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7fe9bffff700 1 -- 172.21.15.103:0/3258663222 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe9c4005bd0 con 0x7fe9b40039d0 2023-07-02T08:59:41.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7f9739d9b700 1 --2- 172.21.15.103:0/234142246 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f971c037730 0x7f971c039c40 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe9cf492700 1 -- 172.21.15.103:0/3258663222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 msgr2=0x7fe9b4003e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe9cf492700 1 --2- 172.21.15.103:0/3258663222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9b4003e20 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe8bc4c8700 1 Processor -- start 2023-07-02T08:59:41.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.037+0000 7fe8bc4c8700 1 -- start start 2023-07-02T08:59:41.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7fe8bc4c8700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe8b409e3c0 0x7fe8b41358a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7fe8bc4c8700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b409b590 con 0x7fe8b409d8a0 2023-07-02T08:59:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.038+0000 7fe8bc4c8700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b409a5b0 con 0x7fe8b409e3c0 2023-07-02T08:59:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe8b9a63700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe8b409e3c0 0x7fe8b41358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe8b9a63700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe8b409e3c0 0x7fe8b41358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55204/0 (socket says 172.21.15.103:55204) 2023-07-02T08:59:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe8b9a63700 1 -- 172.21.15.103:0/3195542266 learned_addr learned my addr 172.21.15.103:0/3195542266 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe8b9a63700 1 -- 172.21.15.103:0/3195542266 >> v1:172.21.15.103:6789/0 conn(0x7fe8b409d8a0 legacy=0x7fe8b409dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe8b9a63700 1 -- 172.21.15.103:0/3195542266 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b409f700 con 0x7fe8b409e3c0 2023-07-02T08:59:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe8b9a63700 1 --2- 172.21.15.103:0/3195542266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409e3c0 0x7fe8b41358a0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd327db206ed3ab6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe9cf492700 1 -- 172.21.15.103:0/3258663222 shutdown_connections 2023-07-02T08:59:41.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe9cf492700 1 --2- 172.21.15.103:0/3258663222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9b4003e20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe9cf492700 1 --2- 172.21.15.103:0/3258663222 >> v2:172.21.15.103:3300/0 conn(0x7fe9c809e3c0 0x7fe9c811b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.039+0000 7fe9cf492700 1 -- 172.21.15.103:0/3258663222 >> 172.21.15.103:0/3258663222 conn(0x7fe9c80a3280 msgr2=0x7fe9c80a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.040+0000 7fe8b9262700 1 -- 172.21.15.103:0/3195542266 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe8a4009d90 con 0x7fe8b409e3c0 2023-07-02T08:59:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.040+0000 7fe8b9262700 1 -- 172.21.15.103:0/3195542266 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe8a400e230 con 0x7fe8b409e3c0 2023-07-02T08:59:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.040+0000 7fe8b9262700 1 -- 172.21.15.103:0/3195542266 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe8a400e4d0 con 0x7fe8b409e3c0 2023-07-02T08:59:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.040+0000 7fe9cf492700 1 -- 172.21.15.103:0/3258663222 shutdown_connections 2023-07-02T08:59:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.040+0000 7f973ad9d700 1 --2- 172.21.15.103:0/234142246 >> v2:172.21.15.103:6800/3256834854 conn(0x7f973c1bb260 0x7f973c12c3e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.040+0000 7faf6ffff700 1 --2- 172.21.15.103:0/3955600816 >> v2:172.21.15.164:6802/3618079103 conn(0x7faf5c03c770 0x7faf5c03ec00 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/3195542266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409e3c0 msgr2=0x7fe8b41358a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe8bc4c8700 1 --2- 172.21.15.103:0/3195542266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409e3c0 0x7fe8b41358a0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.040+0000 7fe9cf492700 1 -- 172.21.15.103:0/3258663222 wait complete. 2023-07-02T08:59:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.040+0000 7fe9cf492700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe9cf492700 1 Processor -- start 2023-07-02T08:59:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/3195542266 shutdown_connections 2023-07-02T08:59:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe8bc4c8700 1 --2- 172.21.15.103:0/3195542266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409e3c0 0x7fe8b41358a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/3195542266 >> 172.21.15.103:0/3195542266 conn(0x7fe8b40a3280 msgr2=0x7fe8b40a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/3195542266 shutdown_connections 2023-07-02T08:59:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/3195542266 wait complete. 2023-07-02T08:59:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe8bc4c8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7f972affd700 1 -- 172.21.15.103:0/234142246 <== osd.0 v2:172.21.15.103:6800/3256834854 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+4964 (crc 0 0 0) 0x7f973400a870 con 0x7f973c1bb260 2023-07-02T08:59:41.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 <== osd.3 v2:172.21.15.164:6802/3618079103 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+4964 (crc 0 0 0) 0x7faf5c03f290 con 0x7faf5c03c770 2023-07-02T08:59:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.041+0000 7fe9cf492700 1 -- start start 2023-07-02T08:59:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7fe9cf492700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9c81cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7fe9cf492700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c811eb90 con 0x7fe9b40039d0 2023-07-02T08:59:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7fe9cd22e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9c81cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7fe9cd22e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9c81cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55216/0 (socket says 172.21.15.103:55216) 2023-07-02T08:59:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7fe9cd22e700 1 -- 172.21.15.103:0/1491921186 learned_addr learned my addr 172.21.15.103:0/1491921186 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7fe9cd22e700 1 -- 172.21.15.103:0/1491921186 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c811f4f0 con 0x7fe9b40039d0 2023-07-02T08:59:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7fe9cd22e700 1 --2- 172.21.15.103:0/1491921186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9c81cb220 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.042+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe9c40071e0 con 0x7fe9b40039d0 2023-07-02T08:59:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.043+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe9c4007340 con 0x7fe9b40039d0 2023-07-02T08:59:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.043+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe9c40075e0 con 0x7fe9b40039d0 2023-07-02T08:59:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.043+0000 7fe8bc4c8700 1 Processor -- start 2023-07-02T08:59:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.043+0000 7fe8bc4c8700 1 -- start start 2023-07-02T08:59:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.044+0000 7fe8bc4c8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409d8a0 0x7fe8b406d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.044+0000 7fe8bc4c8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b41a8650 con 0x7fe8b409d8a0 2023-07-02T08:59:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.044+0000 7fe8ba264700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409d8a0 0x7fe8b406d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.044+0000 7fe8ba264700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409d8a0 0x7fe8b406d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55224/0 (socket says 172.21.15.103:55224) 2023-07-02T08:59:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.044+0000 7fe8ba264700 1 -- 172.21.15.103:0/857042296 learned_addr learned my addr 172.21.15.103:0/857042296 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.044+0000 7fe8ba264700 1 -- 172.21.15.103:0/857042296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b41a91f0 con 0x7fe8b409d8a0 2023-07-02T08:59:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.044+0000 7fe8ba264700 1 --2- 172.21.15.103:0/857042296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409d8a0 0x7fe8b406d0a0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.045+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe8b000c8a0 con 0x7fe8b409d8a0 2023-07-02T08:59:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.045+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe8b000ca00 con 0x7fe8b409d8a0 2023-07-02T08:59:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.045+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b41a8ca0 con 0x7fe8b409d8a0 2023-07-02T08:59:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.045+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe8b000cca0 con 0x7fe8b409d8a0 2023-07-02T08:59:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.045+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b41b80a0 con 0x7fe8b409d8a0 2023-07-02T08:59:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.045+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7fe8b0011310 con 0x7fe8b409d8a0 2023-07-02T08:59:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.045+0000 7fe8aaffd700 1 --2- 172.21.15.103:0/857042296 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8980376e0 0x7fe898039bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.046+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fe8b0047120 con 0x7fe8b409d8a0 2023-07-02T08:59:41.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.046+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8b41a8b20 con 0x7fe8b409d8a0 2023-07-02T08:59:41.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.046+0000 7fe8aaffd700 1 --2- 172.21.15.103:0/857042296 >> v2:172.21.15.164:6800/4222444210 conn(0x7fe89803c680 0x7fe89803eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.046+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 --> v2:172.21.15.164:6800/4222444210 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe89803f1a0 con 0x7fe89803c680 2023-07-02T08:59:41.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.046+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (crc 0 0 0) 0x7fe8b00102a0 con 0x7fe8b409d8a0 2023-07-02T08:59:41.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.046+0000 7fe8b9a63700 1 --2- 172.21.15.103:0/857042296 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8980376e0 0x7fe898039bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.046+0000 7fe8baa65700 1 --2- 172.21.15.103:0/857042296 >> v2:172.21.15.164:6800/4222444210 conn(0x7fe89803c680 0x7fe89803eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-07-02T08:59:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.043+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c8120220 con 0x7fe9b40039d0 2023-07-02T08:59:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe8b9a63700 1 --2- 172.21.15.103:0/857042296 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8980376e0 0x7fe898039bf0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 --> v2:172.21.15.103:6800/3256834854 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f973c06f490 con 0x7f973c1bb260 2023-07-02T08:59:41.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.043+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c8120c30 con 0x7fe9b40039d0 2023-07-02T08:59:41.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9c8121b70 con 0x7fe9b40039d0 2023-07-02T08:59:41.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 44703+0+0 (crc 0 0 0) 0x7fe9c40136b0 con 0x7fe9b40039d0 2023-07-02T08:59:41.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe9bdffb700 1 --2- 172.21.15.103:0/1491921186 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9ac0376e0 0x7fe9ac039bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fe9c4047fc0 con 0x7fe9b40039d0 2023-07-02T08:59:41.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe9bdffb700 1 --2- 172.21.15.103:0/1491921186 >> v2:172.21.15.103:6802/1949274160 conn(0x7fe9ac03c680 0x7fe9ac03eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 --> v2:172.21.15.103:6802/1949274160 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9ac03f1a0 con 0x7fe9ac03c680 2023-07-02T08:59:41.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (crc 0 0 0) 0x7fe9c404a420 con 0x7fe9b40039d0 2023-07-02T08:59:41.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.047+0000 7fe9cca2d700 1 --2- 172.21.15.103:0/1491921186 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9ac0376e0 0x7fe9ac039bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.048+0000 7fe9cca2d700 1 --2- 172.21.15.103:0/1491921186 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9ac0376e0 0x7fe9ac039bf0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.048+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 --> v2:172.21.15.164:6802/3618079103 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faf50002b20 con 0x7faf5c03c770 2023-07-02T08:59:41.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.048+0000 7fe9cda2f700 1 --2- 172.21.15.103:0/1491921186 >> v2:172.21.15.103:6802/1949274160 conn(0x7fe9ac03c680 0x7fe9ac03eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-07-02T08:59:41.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.049+0000 7fe8baa65700 1 --2- 172.21.15.103:0/857042296 >> v2:172.21.15.164:6800/4222444210 conn(0x7fe89803c680 0x7fe89803eb10 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.050+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 <== osd.2 v2:172.21.15.164:6800/4222444210 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+4964 (crc 0 0 0) 0x7fe89803f1a0 con 0x7fe89803c680 2023-07-02T08:59:41.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.051+0000 7fe9cda2f700 1 --2- 172.21.15.103:0/1491921186 >> v2:172.21.15.103:6802/1949274160 conn(0x7fe9ac03c680 0x7fe9ac03eb10 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.051+0000 7faf57fff700 1 -- 172.21.15.103:0/3955600816 <== osd.3 v2:172.21.15.164:6802/3618079103 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faf50002b20 con 0x7faf5c03c770 2023-07-02T08:59:41.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 <== osd.1 v2:172.21.15.103:6802/1949274160 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+4964 (crc 0 0 0) 0x7fe9ac03f1a0 con 0x7fe9ac03c680 2023-07-02T08:59:41.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 >> v2:172.21.15.164:6802/3618079103 conn(0x7faf5c03c770 msgr2=0x7faf5c03ec00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 --2- 172.21.15.103:0/3955600816 >> v2:172.21.15.164:6802/3618079103 conn(0x7faf5c03c770 0x7faf5c03ec00 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf5c037780 msgr2=0x7faf5c039c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 --2- 172.21.15.103:0/3955600816 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf5c037780 0x7faf5c039c90 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 msgr2=0x7faf7011c5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 --2- 172.21.15.103:0/3955600816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf7011c5d0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 shutdown_connections 2023-07-02T08:59:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 --2- 172.21.15.103:0/3955600816 >> v2:172.21.15.164:6802/3618079103 conn(0x7faf5c03c770 0x7faf5c03ec00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 --2- 172.21.15.103:0/3955600816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf7009d750 0x7faf7011c5d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 --2- 172.21.15.103:0/3955600816 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf5c037780 0x7faf5c039c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 >> 172.21.15.103:0/3955600816 conn(0x7faf700a1a80 msgr2=0x7faf701ba940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 shutdown_connections 2023-07-02T08:59:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.052+0000 7faf75d40700 1 -- 172.21.15.103:0/3955600816 wait complete. 2023-07-02T08:59:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.053+0000 7f972affd700 1 -- 172.21.15.103:0/234142246 <== osd.0 v2:172.21.15.103:6800/3256834854 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f973c06f490 con 0x7f973c1bb260 2023-07-02T08:59:41.081 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 >> v2:172.21.15.103:6800/3256834854 conn(0x7f973c1bb260 msgr2=0x7f973c12c3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.081 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 --2- 172.21.15.103:0/234142246 >> v2:172.21.15.103:6800/3256834854 conn(0x7f973c1bb260 0x7f973c12c3e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.081 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 --> v2:172.21.15.164:6800/4222444210 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe8b406ec20 con 0x7fe89803c680 2023-07-02T08:59:41.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f971c037730 msgr2=0x7f971c039c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 --2- 172.21.15.103:0/234142246 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f971c037730 0x7f971c039c40 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c132f50 msgr2=0x7f973c12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 --2- 172.21.15.103:0/234142246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c132f50 0x7f973c12bea0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 shutdown_connections 2023-07-02T08:59:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 --2- 172.21.15.103:0/234142246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f973c132f50 0x7f973c12bea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 --2- 172.21.15.103:0/234142246 >> v2:172.21.15.103:6800/3256834854 conn(0x7f973c1bb260 0x7f973c12c3e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 --2- 172.21.15.103:0/234142246 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f971c037730 0x7f971c039c40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 >> 172.21.15.103:0/234142246 conn(0x7f973c0a2080 msgr2=0x7f973c1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 shutdown_connections 2023-07-02T08:59:41.084 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.054+0000 7f974097a700 1 -- 172.21.15.103:0/234142246 wait complete. 2023-07-02T08:59:41.084 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8aaffd700 1 -- 172.21.15.103:0/857042296 <== osd.2 v2:172.21.15.164:6800/4222444210 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe8b406ec20 con 0x7fe89803c680 2023-07-02T08:59:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 >> v2:172.21.15.164:6800/4222444210 conn(0x7fe89803c680 msgr2=0x7fe89803eb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 --2- 172.21.15.103:0/857042296 >> v2:172.21.15.164:6800/4222444210 conn(0x7fe89803c680 0x7fe89803eb10 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8980376e0 msgr2=0x7fe898039bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 --2- 172.21.15.103:0/857042296 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8980376e0 0x7fe898039bf0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409d8a0 msgr2=0x7fe8b406d0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 --2- 172.21.15.103:0/857042296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409d8a0 0x7fe8b406d0a0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 shutdown_connections 2023-07-02T08:59:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 --2- 172.21.15.103:0/857042296 >> v2:172.21.15.164:6800/4222444210 conn(0x7fe89803c680 0x7fe89803eb10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 --2- 172.21.15.103:0/857042296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b409d8a0 0x7fe8b406d0a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 --2- 172.21.15.103:0/857042296 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8980376e0 0x7fe898039bf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.055+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 >> 172.21.15.103:0/857042296 conn(0x7fe8b40a3280 msgr2=0x7fe8b41b85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.056+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 shutdown_connections 2023-07-02T08:59:41.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.056+0000 7fe8bc4c8700 1 -- 172.21.15.103:0/857042296 wait complete. 2023-07-02T08:59:41.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.057+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 --> v2:172.21.15.103:6802/1949274160 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9c809a5b0 con 0x7fe9ac03c680 2023-07-02T08:59:41.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.058+0000 7fe9bdffb700 1 -- 172.21.15.103:0/1491921186 <== osd.1 v2:172.21.15.103:6802/1949274160 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe9c000a870 con 0x7fe9ac03c680 2023-07-02T08:59:41.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.058+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 >> v2:172.21.15.103:6802/1949274160 conn(0x7fe9ac03c680 msgr2=0x7fe9ac03eb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.058+0000 7fe9cf492700 1 --2- 172.21.15.103:0/1491921186 >> v2:172.21.15.103:6802/1949274160 conn(0x7fe9ac03c680 0x7fe9ac03eb10 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.058+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9ac0376e0 msgr2=0x7fe9ac039bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.058+0000 7fe9cf492700 1 --2- 172.21.15.103:0/1491921186 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9ac0376e0 0x7fe9ac039bf0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.058+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 msgr2=0x7fe9c81cb220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.058+0000 7fe9cf492700 1 --2- 172.21.15.103:0/1491921186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9c81cb220 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.065+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 shutdown_connections 2023-07-02T08:59:41.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.065+0000 7fe9cf492700 1 --2- 172.21.15.103:0/1491921186 >> v2:172.21.15.103:6802/1949274160 conn(0x7fe9ac03c680 0x7fe9ac03eb10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.065+0000 7fe9cf492700 1 --2- 172.21.15.103:0/1491921186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9b40039d0 0x7fe9c81cb220 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.065+0000 7fe9cf492700 1 --2- 172.21.15.103:0/1491921186 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9ac0376e0 0x7fe9ac039bf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.065+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 >> 172.21.15.103:0/1491921186 conn(0x7fe9c80a3280 msgr2=0x7fe9c806f510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.065+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 shutdown_connections 2023-07-02T08:59:41.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.065+0000 7fe9cf492700 1 -- 172.21.15.103:0/1491921186 wait complete. 2023-07-02T08:59:41.090 INFO:teuthology.orchestra.run.smithi103.stdout:38654705670 2023-07-02T08:59:41.091 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:41.091 INFO:teuthology.orchestra.run.smithi103.stdout:38654705670 2023-07-02T08:59:41.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:41.092 INFO:teuthology.orchestra.run.smithi103.stdout:38654705670 2023-07-02T08:59:41.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:41.094 INFO:teuthology.orchestra.run.smithi103.stdout:38654705670 2023-07-02T08:59:41.094 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:41.284 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.274+0000 7efdb79d4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.285 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.284+0000 7efdb79d4700 1 Processor -- start 2023-07-02T08:59:41.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.284+0000 7efdb79d4700 1 -- start start 2023-07-02T08:59:41.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.285+0000 7efdb79d4700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdb0132f50 0x7efdb0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.285+0000 7efdb79d4700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb009e260 con 0x7efdb009c580 2023-07-02T08:59:41.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.285+0000 7efdb79d4700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00a1640 con 0x7efdb0132f50 2023-07-02T08:59:41.288 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.286+0000 7efdb5770700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdb0132f50 0x7efdb0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.288 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.286+0000 7efdb4f6f700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7efdb009c580 0x7efdb009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43112/0 (socket says 172.21.15.103:43112) 2023-07-02T08:59:41.288 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efdb4f6f700 1 -- 172.21.15.103:0/4228809888 learned_addr learned my addr 172.21.15.103:0/4228809888 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:41.288 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.281+0000 7f85e5721700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efda7fff700 1 -- 172.21.15.103:0/4228809888 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469082905 0 0) 0x7efdb009e260 con 0x7efdb009c580 2023-07-02T08:59:41.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efda7fff700 1 -- 172.21.15.103:0/4228809888 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c003580 con 0x7efdb009c580 2023-07-02T08:59:41.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efdb5770700 1 -- 172.21.15.103:0/4228809888 >> v1:172.21.15.103:6789/0 conn(0x7efdb009c580 legacy=0x7efdb009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efdb5770700 1 -- 172.21.15.103:0/4228809888 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb0132a20 con 0x7efdb0132f50 2023-07-02T08:59:41.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efdb5770700 1 --2- 172.21.15.103:0/4228809888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb0132f50 0x7efdb0133360 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1d3aad937b78df server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efda7fff700 1 -- 172.21.15.103:0/4228809888 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdac009c80 con 0x7efdb0132f50 2023-07-02T08:59:41.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efda7fff700 1 -- 172.21.15.103:0/4228809888 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efdac009de0 con 0x7efdb0132f50 2023-07-02T08:59:41.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.287+0000 7efda7fff700 1 -- 172.21.15.103:0/4228809888 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdac00e360 con 0x7efdb0132f50 2023-07-02T08:59:41.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.288+0000 7f85e5721700 1 Processor -- start 2023-07-02T08:59:41.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.288+0000 7f85e5721700 1 -- start start 2023-07-02T08:59:41.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.288+0000 7efdb79d4700 1 -- 172.21.15.103:0/4228809888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb0132f50 msgr2=0x7efdb0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.288+0000 7efdb79d4700 1 --2- 172.21.15.103:0/4228809888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb0132f50 0x7efdb0133360 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.288+0000 7efdb79d4700 1 -- 172.21.15.103:0/4228809888 shutdown_connections 2023-07-02T08:59:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.288+0000 7efdb79d4700 1 --2- 172.21.15.103:0/4228809888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb0132f50 0x7efdb0133360 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.288+0000 7efdb79d4700 1 -- 172.21.15.103:0/4228809888 >> 172.21.15.103:0/4228809888 conn(0x7efdb00a2080 msgr2=0x7efdb00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7efdb79d4700 1 -- 172.21.15.103:0/4228809888 shutdown_connections 2023-07-02T08:59:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7efdb79d4700 1 -- 172.21.15.103:0/4228809888 wait complete. 2023-07-02T08:59:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7f85e5721700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f85e009d8a0 0x7f85e009dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7f85e5721700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e009b590 con 0x7f85e009e1f0 2023-07-02T08:59:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7f85e5721700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e009a5b0 con 0x7f85e009d8a0 2023-07-02T08:59:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7efdb79d4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7f85de7fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f85e009e1f0 0x7f85e0130660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43114/0 (socket says 172.21.15.103:43114) 2023-07-02T08:59:41.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7f85de7fc700 1 -- 172.21.15.103:0/384951553 learned_addr learned my addr 172.21.15.103:0/384951553 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:41.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.289+0000 7f85deffd700 1 --2- 172.21.15.103:0/384951553 >> v2:172.21.15.103:3300/0 conn(0x7f85e009d8a0 0x7f85e009dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.290+0000 7f85ddffb700 1 -- 172.21.15.103:0/384951553 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083553950 0 0) 0x7f85e009b590 con 0x7f85e009e1f0 2023-07-02T08:59:41.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.290+0000 7f85ddffb700 1 -- 172.21.15.103:0/384951553 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c8003580 con 0x7f85e009e1f0 2023-07-02T08:59:41.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.290+0000 7f85deffd700 1 -- 172.21.15.103:0/384951553 >> v1:172.21.15.103:6789/0 conn(0x7f85e009e1f0 legacy=0x7f85e0130660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.290+0000 7f85deffd700 1 -- 172.21.15.103:0/384951553 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e009f700 con 0x7f85e009d8a0 2023-07-02T08:59:41.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.290+0000 7efdb79d4700 1 Processor -- start 2023-07-02T08:59:41.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.290+0000 7efdb79d4700 1 -- start start 2023-07-02T08:59:41.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.284+0000 7f903829a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7f85deffd700 1 --2- 172.21.15.103:0/384951553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e009dcb0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd65319827c7e4a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7f85ddffb700 1 -- 172.21.15.103:0/384951553 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f85d0009c80 con 0x7f85e009d8a0 2023-07-02T08:59:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7f85ddffb700 1 -- 172.21.15.103:0/384951553 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f85d0009de0 con 0x7f85e009d8a0 2023-07-02T08:59:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7f85ddffb700 1 -- 172.21.15.103:0/384951553 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f85d000e360 con 0x7f85e009d8a0 2023-07-02T08:59:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7efdb79d4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb009c580 0x7efdb012c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7efdb79d4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb008b8d0 con 0x7efdb009c580 2023-07-02T08:59:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7f903829a700 1 Processor -- start 2023-07-02T08:59:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7efdb5770700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb009c580 0x7efdb012c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7efdb5770700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb009c580 0x7efdb012c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55238/0 (socket says 172.21.15.103:55238) 2023-07-02T08:59:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7efdb5770700 1 -- 172.21.15.103:0/1362675276 learned_addr learned my addr 172.21.15.103:0/1362675276 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.292+0000 7f85e5721700 1 -- 172.21.15.103:0/384951553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 msgr2=0x7f85e009dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.292+0000 7f85e5721700 1 --2- 172.21.15.103:0/384951553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e009dcb0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.291+0000 7f903829a700 1 -- start start 2023-07-02T08:59:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.292+0000 7f903829a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9030133360 0x7f903009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.292+0000 7f903829a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903009e260 con 0x7f9030132f50 2023-07-02T08:59:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.292+0000 7f903829a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90300a1640 con 0x7f9030133360 2023-07-02T08:59:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.292+0000 7f9036036700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9030132f50 0x7f903009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43130/0 (socket says 172.21.15.103:43130) 2023-07-02T08:59:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.292+0000 7f9036036700 1 -- 172.21.15.103:0/1639490331 learned_addr learned my addr 172.21.15.103:0/1639490331 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f9035835700 1 --2- 172.21.15.103:0/1639490331 >> v2:172.21.15.103:3300/0 conn(0x7f9030133360 0x7f903009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7efdb5770700 1 -- 172.21.15.103:0/1362675276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb008db80 con 0x7efdb009c580 2023-07-02T08:59:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f9035034700 1 -- 172.21.15.103:0/1639490331 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780200610 0 0) 0x7f903009e260 con 0x7f9030132f50 2023-07-02T08:59:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f9035034700 1 -- 172.21.15.103:0/1639490331 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f901c003580 con 0x7f9030132f50 2023-07-02T08:59:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f85e5721700 1 -- 172.21.15.103:0/384951553 shutdown_connections 2023-07-02T08:59:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f85e5721700 1 --2- 172.21.15.103:0/384951553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e009dcb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f85e5721700 1 -- 172.21.15.103:0/384951553 >> 172.21.15.103:0/384951553 conn(0x7f85e00a3280 msgr2=0x7f85e00a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f85e5721700 1 -- 172.21.15.103:0/384951553 shutdown_connections 2023-07-02T08:59:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f85e5721700 1 -- 172.21.15.103:0/384951553 wait complete. 2023-07-02T08:59:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f9035835700 1 -- 172.21.15.103:0/1639490331 >> v1:172.21.15.103:6789/0 conn(0x7f9030132f50 legacy=0x7f903009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7f9035835700 1 -- 172.21.15.103:0/1639490331 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030132a20 con 0x7f9030133360 2023-07-02T08:59:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7f9035835700 1 --2- 172.21.15.103:0/1639490331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030133360 0x7f903009cc90 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41e8528cef990dd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.293+0000 7efdb5770700 1 --2- 172.21.15.103:0/1362675276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb009c580 0x7efdb012c700 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7efda5ffb700 1 -- 172.21.15.103:0/1362675276 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdac014540 con 0x7efdb009c580 2023-07-02T08:59:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7efda5ffb700 1 -- 172.21.15.103:0/1362675276 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efdac0146a0 con 0x7efdb009c580 2023-07-02T08:59:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb008a1e0 con 0x7efdb009c580 2023-07-02T08:59:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7efda5ffb700 1 -- 172.21.15.103:0/1362675276 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdac014940 con 0x7efdb009c580 2023-07-02T08:59:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7f9035034700 1 -- 172.21.15.103:0/1639490331 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9020009c80 con 0x7f9030133360 2023-07-02T08:59:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7f9035034700 1 -- 172.21.15.103:0/1639490331 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9020009de0 con 0x7f9030133360 2023-07-02T08:59:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7f9035034700 1 -- 172.21.15.103:0/1639490331 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f902000e360 con 0x7f9030133360 2023-07-02T08:59:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7f85e5721700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.295+0000 7f85e5721700 1 Processor -- start 2023-07-02T08:59:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.294+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb008b520 con 0x7efdb009c580 2023-07-02T08:59:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.295+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb0061ea0 con 0x7efdb009c580 2023-07-02T08:59:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.295+0000 7f903829a700 1 -- 172.21.15.103:0/1639490331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030133360 msgr2=0x7f903009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.295+0000 7f903829a700 1 --2- 172.21.15.103:0/1639490331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030133360 0x7f903009cc90 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7efda5ffb700 1 -- 172.21.15.103:0/1362675276 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7efdac014aa0 con 0x7efdb009c580 2023-07-02T08:59:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7f903829a700 1 -- 172.21.15.103:0/1639490331 shutdown_connections 2023-07-02T08:59:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7f903829a700 1 --2- 172.21.15.103:0/1639490331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030133360 0x7f903009cc90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7f903829a700 1 -- 172.21.15.103:0/1639490331 >> 172.21.15.103:0/1639490331 conn(0x7f90300a2080 msgr2=0x7f90300a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7f903829a700 1 -- 172.21.15.103:0/1639490331 shutdown_connections 2023-07-02T08:59:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7f85e5721700 1 -- start start 2023-07-02T08:59:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.297+0000 7f85e5721700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.297+0000 7f85e5721700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e006ecb0 con 0x7f85e009d8a0 2023-07-02T08:59:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.298+0000 7f85deffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.298+0000 7f85deffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e006da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55258/0 (socket says 172.21.15.103:55258) 2023-07-02T08:59:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.298+0000 7f85deffd700 1 -- 172.21.15.103:0/3030473565 learned_addr learned my addr 172.21.15.103:0/3030473565 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7f903829a700 1 -- 172.21.15.103:0/1639490331 wait complete. 2023-07-02T08:59:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.297+0000 7f903829a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.298+0000 7f903829a700 1 Processor -- start 2023-07-02T08:59:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.298+0000 7f85deffd700 1 -- 172.21.15.103:0/3030473565 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e006d5c0 con 0x7f85e009d8a0 2023-07-02T08:59:41.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.298+0000 7f85deffd700 1 --2- 172.21.15.103:0/3030473565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e006da10 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.298+0000 7f903829a700 1 -- start start 2023-07-02T08:59:41.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.299+0000 7f903829a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030132f50 0x7f9030089f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.299+0000 7f903829a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903008b170 con 0x7f9030132f50 2023-07-02T08:59:41.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7efda5ffb700 1 --2- 172.21.15.103:0/1362675276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd940377c0 0x7efd94039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.296+0000 7efda5ffb700 1 -- 172.21.15.103:0/1362675276 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7efdac048210 con 0x7efdb009c580 2023-07-02T08:59:41.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.297+0000 7efdb4f6f700 1 --2- 172.21.15.103:0/1362675276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd940377c0 0x7efd94039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.297+0000 7efdb4f6f700 1 --2- 172.21.15.103:0/1362675276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd940377c0 0x7efd94039cd0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.300+0000 7f9036036700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030132f50 0x7f9030089f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.300+0000 7f9036036700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030132f50 0x7f9030089f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55266/0 (socket says 172.21.15.103:55266) 2023-07-02T08:59:41.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.300+0000 7f9036036700 1 -- 172.21.15.103:0/1322694799 learned_addr learned my addr 172.21.15.103:0/1322694799 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.300+0000 7f9036036700 1 -- 172.21.15.103:0/1322694799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030133a70 con 0x7f9030132f50 2023-07-02T08:59:41.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.300+0000 7f9036036700 1 --2- 172.21.15.103:0/1322694799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030132f50 0x7f9030089f90 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f85c77fe700 1 -- 172.21.15.103:0/3030473565 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f85d0013530 con 0x7f85e009d8a0 2023-07-02T08:59:41.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f85c77fe700 1 -- 172.21.15.103:0/3030473565 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f85d0013690 con 0x7f85e009d8a0 2023-07-02T08:59:41.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f85c77fe700 1 -- 172.21.15.103:0/3030473565 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f85d0013930 con 0x7f85e009d8a0 2023-07-02T08:59:41.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7efda5ffb700 1 -- 172.21.15.103:0/1362675276 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7efdac00fb20 con 0x7efdb009c580 2023-07-02T08:59:41.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f9026ffd700 1 -- 172.21.15.103:0/1322694799 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f902c005a60 con 0x7f9030132f50 2023-07-02T08:59:41.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f9026ffd700 1 -- 172.21.15.103:0/1322694799 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f902c005bc0 con 0x7f9030132f50 2023-07-02T08:59:41.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f9026ffd700 1 -- 172.21.15.103:0/1322694799 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f902c005e60 con 0x7f9030132f50 2023-07-02T08:59:41.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e01b9160 con 0x7f85e009d8a0 2023-07-02T08:59:41.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e006d220 con 0x7f85e009d8a0 2023-07-02T08:59:41.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903008b9c0 con 0x7f9030132f50 2023-07-02T08:59:41.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.303+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903008dcb0 con 0x7f9030132f50 2023-07-02T08:59:41.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.304+0000 7f9026ffd700 1 -- 172.21.15.103:0/1322694799 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f902c0070b0 con 0x7f9030132f50 2023-07-02T08:59:41.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.304+0000 7f9026ffd700 1 --2- 172.21.15.103:0/1322694799 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9014037810 0x7f9014039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.304+0000 7f9026ffd700 1 -- 172.21.15.103:0/1322694799 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f902c046a20 con 0x7f9030132f50 2023-07-02T08:59:41.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.305+0000 7f9035835700 1 --2- 172.21.15.103:0/1322694799 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9014037810 0x7f9014039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.305+0000 7f85c77fe700 1 -- 172.21.15.103:0/3030473565 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f85d0013a90 con 0x7f85e009d8a0 2023-07-02T08:59:41.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.305+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c00050b0 con 0x7f85e009d8a0 2023-07-02T08:59:41.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.305+0000 7f85c77fe700 1 --2- 172.21.15.103:0/3030473565 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f85cc0377c0 0x7f85cc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.305+0000 7f85c77fe700 1 -- 172.21.15.103:0/3030473565 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f85d0047f10 con 0x7f85e009d8a0 2023-07-02T08:59:41.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.305+0000 7f9035835700 1 --2- 172.21.15.103:0/1322694799 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9014037810 0x7f9014039d20 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.305+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9030133a70 con 0x7f9030132f50 2023-07-02T08:59:41.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.306+0000 7f85de7fc700 1 --2- 172.21.15.103:0/3030473565 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f85cc0377c0 0x7f85cc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.306+0000 7f85de7fc700 1 --2- 172.21.15.103:0/3030473565 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f85cc0377c0 0x7f85cc039cd0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.310+0000 7f85c77fe700 1 -- 172.21.15.103:0/3030473565 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f85d000d760 con 0x7f85e009d8a0 2023-07-02T08:59:41.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.315+0000 7f9026ffd700 1 -- 172.21.15.103:0/1322694799 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f9030133a70 con 0x7f9030132f50 2023-07-02T08:59:41.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.325+0000 7fa1a4fe4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.331+0000 7fa1a4fe4700 1 Processor -- start 2023-07-02T08:59:41.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.331+0000 7fa1a4fe4700 1 -- start start 2023-07-02T08:59:41.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.332+0000 7fa1a4fe4700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1a0132f50 0x7fa1a0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.332+0000 7fa1a4fe4700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a009e260 con 0x7fa1a009c580 2023-07-02T08:59:41.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.332+0000 7fa1a4fe4700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a00a1640 con 0x7fa1a0132f50 2023-07-02T08:59:41.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.332+0000 7fa19e59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1a0132f50 0x7fa1a0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.332+0000 7fa19dd9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa1a009c580 0x7fa1a009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43136/0 (socket says 172.21.15.103:43136) 2023-07-02T08:59:41.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.332+0000 7fa19dd9b700 1 -- 172.21.15.103:0/4283209786 learned_addr learned my addr 172.21.15.103:0/4283209786 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:41.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.332+0000 7fa19e59c700 1 -- 172.21.15.103:0/4283209786 >> v1:172.21.15.103:6789/0 conn(0x7fa1a009c580 legacy=0x7fa1a009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.332+0000 7fa19e59c700 1 -- 172.21.15.103:0/4283209786 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a0132a20 con 0x7fa1a0132f50 2023-07-02T08:59:41.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.333+0000 7fa19e59c700 1 --2- 172.21.15.103:0/4283209786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a0132f50 0x7fa1a0133360 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e42c3ba4978ead server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.333+0000 7fa19d59a700 1 -- 172.21.15.103:0/4283209786 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa190009cd0 con 0x7fa1a0132f50 2023-07-02T08:59:41.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.333+0000 7fa19d59a700 1 -- 172.21.15.103:0/4283209786 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa190009e30 con 0x7fa1a0132f50 2023-07-02T08:59:41.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.333+0000 7fa19d59a700 1 -- 172.21.15.103:0/4283209786 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa19000e360 con 0x7fa1a0132f50 2023-07-02T08:59:41.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.334+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/4283209786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a0132f50 msgr2=0x7fa1a0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.334+0000 7fa1a4fe4700 1 --2- 172.21.15.103:0/4283209786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a0132f50 0x7fa1a0133360 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.334+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/4283209786 shutdown_connections 2023-07-02T08:59:41.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.334+0000 7fa1a4fe4700 1 --2- 172.21.15.103:0/4283209786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a0132f50 0x7fa1a0133360 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.334+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/4283209786 >> 172.21.15.103:0/4283209786 conn(0x7fa1a00a2080 msgr2=0x7fa1a00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.334+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/4283209786 shutdown_connections 2023-07-02T08:59:41.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.334+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/4283209786 wait complete. 2023-07-02T08:59:41.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.334+0000 7fa1a4fe4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:41.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.335+0000 7fa1a4fe4700 1 Processor -- start 2023-07-02T08:59:41.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.336+0000 7fa1a4fe4700 1 -- start start 2023-07-02T08:59:41.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.336+0000 7fa1a4fe4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009c580 0x7fa1a012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.336+0000 7fa1a4fe4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a006f9d0 con 0x7fa1a009c580 2023-07-02T08:59:41.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.336+0000 7fa19e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009c580 0x7fa1a012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.336+0000 7fa19e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009c580 0x7fa1a012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55278/0 (socket says 172.21.15.103:55278) 2023-07-02T08:59:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.336+0000 7fa19e59c700 1 -- 172.21.15.103:0/2341212589 learned_addr learned my addr 172.21.15.103:0/2341212589 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.337+0000 7fa19e59c700 1 -- 172.21.15.103:0/2341212589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a0133a00 con 0x7fa1a009c580 2023-07-02T08:59:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.337+0000 7fa19e59c700 1 --2- 172.21.15.103:0/2341212589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009c580 0x7fa1a012b7b0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.337+0000 7fa18effd700 1 -- 172.21.15.103:0/2341212589 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa190013530 con 0x7fa1a009c580 2023-07-02T08:59:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.337+0000 7fa18effd700 1 -- 172.21.15.103:0/2341212589 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa190013690 con 0x7fa1a009c580 2023-07-02T08:59:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.337+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a006d160 con 0x7fa1a009c580 2023-07-02T08:59:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.337+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a006da70 con 0x7fa1a009c580 2023-07-02T08:59:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.337+0000 7fa18effd700 1 -- 172.21.15.103:0/2341212589 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa190013930 con 0x7fa1a009c580 2023-07-02T08:59:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.338+0000 7fa18effd700 1 -- 172.21.15.103:0/2341212589 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa190013a90 con 0x7fa1a009c580 2023-07-02T08:59:41.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.339+0000 7fa18effd700 1 --2- 172.21.15.103:0/2341212589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa180037760 0x7fa180039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:41.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.339+0000 7fa18effd700 1 -- 172.21.15.103:0/2341212589 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa190018040 con 0x7fa1a009c580 2023-07-02T08:59:41.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.339+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a0061ea0 con 0x7fa1a009c580 2023-07-02T08:59:41.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.339+0000 7fa19dd9b700 1 --2- 172.21.15.103:0/2341212589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa180037760 0x7fa180039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:41.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.340+0000 7fa19dd9b700 1 --2- 172.21.15.103:0/2341212589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa180037760 0x7fa180039c70 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:41.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.349+0000 7fa18effd700 1 -- 172.21.15.103:0/2341212589 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa190018e00 con 0x7fa1a009c580 2023-07-02T08:59:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.514+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f85c0005360 con 0x7f85e009d8a0 2023-07-02T08:59:41.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.514+0000 7f85c77fe700 1 -- 172.21.15.103:0/3030473565 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f85d000fe00 con 0x7f85e009d8a0 2023-07-02T08:59:41.516 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:41.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f85cc0377c0 msgr2=0x7f85cc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 --2- 172.21.15.103:0/3030473565 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f85cc0377c0 0x7f85cc039cd0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 msgr2=0x7f85e006da10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 --2- 172.21.15.103:0/3030473565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e006da10 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 shutdown_connections 2023-07-02T08:59:41.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 --2- 172.21.15.103:0/3030473565 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f85cc0377c0 0x7f85cc039cd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 --2- 172.21.15.103:0/3030473565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e009d8a0 0x7f85e006da10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 >> 172.21.15.103:0/3030473565 conn(0x7f85e00a3280 msgr2=0x7f85e019e200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 shutdown_connections 2023-07-02T08:59:41.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.517+0000 7f85e5721700 1 -- 172.21.15.103:0/3030473565 wait complete. 2023-07-02T08:59:41.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:41.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.542+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efdb0073ca0 con 0x7efdb009c580 2023-07-02T08:59:41.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.542+0000 7efda5ffb700 1 -- 172.21.15.103:0/1362675276 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7efdac00f710 con 0x7efdb009c580 2023-07-02T08:59:41.544 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:41.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd940377c0 msgr2=0x7efd94039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 --2- 172.21.15.103:0/1362675276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd940377c0 0x7efd94039cd0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb009c580 msgr2=0x7efdb012c700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 --2- 172.21.15.103:0/1362675276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb009c580 0x7efdb012c700 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 shutdown_connections 2023-07-02T08:59:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 --2- 172.21.15.103:0/1362675276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd940377c0 0x7efd94039cd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 --2- 172.21.15.103:0/1362675276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdb009c580 0x7efdb012c700 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 >> 172.21.15.103:0/1362675276 conn(0x7efdb00a2080 msgr2=0x7efdb0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 shutdown_connections 2023-07-02T08:59:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.546+0000 7efdb79d4700 1 -- 172.21.15.103:0/1362675276 wait complete. 2023-07-02T08:59:41.563 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.562+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9030133a70 con 0x7f9030132f50 2023-07-02T08:59:41.563 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.562+0000 7f9026ffd700 1 -- 172.21.15.103:0/1322694799 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f9030133a70 con 0x7f9030132f50 2023-07-02T08:59:41.563 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:41.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9014037810 msgr2=0x7f9014039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 --2- 172.21.15.103:0/1322694799 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9014037810 0x7f9014039d20 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030132f50 msgr2=0x7f9030089f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 --2- 172.21.15.103:0/1322694799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030132f50 0x7f9030089f90 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 shutdown_connections 2023-07-02T08:59:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 --2- 172.21.15.103:0/1322694799 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9014037810 0x7f9014039d20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 --2- 172.21.15.103:0/1322694799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9030132f50 0x7f9030089f90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 >> 172.21.15.103:0/1322694799 conn(0x7f90300a2080 msgr2=0x7f9030130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 shutdown_connections 2023-07-02T08:59:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.565+0000 7f903829a700 1 -- 172.21.15.103:0/1322694799 wait complete. 2023-07-02T08:59:41.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.726+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa1a00725b0 con 0x7fa1a009c580 2023-07-02T08:59:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.727+0000 7fa18effd700 1 -- 172.21.15.103:0/2341212589 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa190010e60 con 0x7fa1a009c580 2023-07-02T08:59:41.728 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa180037760 msgr2=0x7fa180039c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 --2- 172.21.15.103:0/2341212589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa180037760 0x7fa180039c70 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009c580 msgr2=0x7fa1a012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 --2- 172.21.15.103:0/2341212589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009c580 0x7fa1a012b7b0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 shutdown_connections 2023-07-02T08:59:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 --2- 172.21.15.103:0/2341212589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa180037760 0x7fa180039c70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 --2- 172.21.15.103:0/2341212589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009c580 0x7fa1a012b7b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 >> 172.21.15.103:0/2341212589 conn(0x7fa1a00a2080 msgr2=0x7fa1a01babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 shutdown_connections 2023-07-02T08:59:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:41.730+0000 7fa1a4fe4700 1 -- 172.21.15.103:0/2341212589 wait complete. 2023-07-02T08:59:41.748 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:42.533 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:42.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:42.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:42.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.667+0000 7f6f970b5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:42.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.674+0000 7f6f970b5700 1 Processor -- start 2023-07-02T08:59:42.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.674+0000 7f6f970b5700 1 -- start start 2023-07-02T08:59:42.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.674+0000 7f6f970b5700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f90133360 0x7f6f9009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.674+0000 7f6f970b5700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9009e260 con 0x7f6f90132f50 2023-07-02T08:59:42.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.674+0000 7f6f970b5700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f900a1640 con 0x7f6f90133360 2023-07-02T08:59:42.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f90133360 0x7f6f9009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f94e51700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6f90132f50 0x7f6f9009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43150/0 (socket says 172.21.15.103:43150) 2023-07-02T08:59:42.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f90133360 0x7f6f9009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55280/0 (socket says 172.21.15.103:55280) 2023-07-02T08:59:42.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f94e51700 1 -- 172.21.15.103:0/860148435 learned_addr learned my addr 172.21.15.103:0/860148435 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:42.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8ffff700 1 -- 172.21.15.103:0/860148435 >> v1:172.21.15.103:6789/0 conn(0x7f6f90132f50 legacy=0x7f6f9009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8ffff700 1 -- 172.21.15.103:0/860148435 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90132a20 con 0x7f6f90133360 2023-07-02T08:59:42.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8f7fe700 1 -- 172.21.15.103:0/860148435 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11679087 0 0) 0x7f6f9009e260 con 0x7f6f90132f50 2023-07-02T08:59:42.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8ffff700 1 --2- 172.21.15.103:0/860148435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90133360 0x7f6f9009cc90 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f9f0d6987ebc30 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8f7fe700 1 -- 172.21.15.103:0/860148435 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6f7c009cd0 con 0x7f6f90133360 2023-07-02T08:59:42.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8f7fe700 1 -- 172.21.15.103:0/860148435 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6f7c009e30 con 0x7f6f90133360 2023-07-02T08:59:42.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.675+0000 7f6f8f7fe700 1 -- 172.21.15.103:0/860148435 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6f7c00e360 con 0x7f6f90133360 2023-07-02T08:59:42.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.676+0000 7f6f970b5700 1 -- 172.21.15.103:0/860148435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90133360 msgr2=0x7f6f9009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.676+0000 7f6f970b5700 1 --2- 172.21.15.103:0/860148435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90133360 0x7f6f9009cc90 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.676+0000 7f6f970b5700 1 -- 172.21.15.103:0/860148435 shutdown_connections 2023-07-02T08:59:42.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.676+0000 7f6f970b5700 1 --2- 172.21.15.103:0/860148435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90133360 0x7f6f9009cc90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.676+0000 7f6f970b5700 1 -- 172.21.15.103:0/860148435 >> 172.21.15.103:0/860148435 conn(0x7f6f900a2080 msgr2=0x7f6f900a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:42.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.676+0000 7f6f970b5700 1 -- 172.21.15.103:0/860148435 shutdown_connections 2023-07-02T08:59:42.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.676+0000 7f6f970b5700 1 -- 172.21.15.103:0/860148435 wait complete. 2023-07-02T08:59:42.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.676+0000 7f6f970b5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:42.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.678+0000 7f6f970b5700 1 Processor -- start 2023-07-02T08:59:42.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.678+0000 7f6f970b5700 1 -- start start 2023-07-02T08:59:42.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f970b5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90132f50 0x7f6f9012c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f970b5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9006d2f0 con 0x7f6f90132f50 2023-07-02T08:59:42.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f94e51700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90132f50 0x7f6f9012c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f94e51700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90132f50 0x7f6f9012c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55292/0 (socket says 172.21.15.103:55292) 2023-07-02T08:59:42.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f94e51700 1 -- 172.21.15.103:0/1971339815 learned_addr learned my addr 172.21.15.103:0/1971339815 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:42.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f94e51700 1 -- 172.21.15.103:0/1971339815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9006dd60 con 0x7f6f90132f50 2023-07-02T08:59:42.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f94e51700 1 --2- 172.21.15.103:0/1971339815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90132f50 0x7f6f9012c1a0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f8d7fa700 1 -- 172.21.15.103:0/1971339815 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6f84004b80 con 0x7f6f90132f50 2023-07-02T08:59:42.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f8d7fa700 1 -- 172.21.15.103:0/1971339815 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6f84004ce0 con 0x7f6f90132f50 2023-07-02T08:59:42.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9012bc50 con 0x7f6f90132f50 2023-07-02T08:59:42.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f8d7fa700 1 -- 172.21.15.103:0/1971339815 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6f8400c4f0 con 0x7f6f90132f50 2023-07-02T08:59:42.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.679+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9006f850 con 0x7f6f90132f50 2023-07-02T08:59:42.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.681+0000 7f6f8d7fa700 1 -- 172.21.15.103:0/1971339815 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6f8400c650 con 0x7f6f90132f50 2023-07-02T08:59:42.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.681+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9006dbb0 con 0x7f6f90132f50 2023-07-02T08:59:42.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.681+0000 7f6f8d7fa700 1 --2- 172.21.15.103:0/1971339815 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6f740377c0 0x7f6f74039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.681+0000 7f6f8d7fa700 1 -- 172.21.15.103:0/1971339815 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6f84046970 con 0x7f6f90132f50 2023-07-02T08:59:42.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.681+0000 7f6f8ffff700 1 --2- 172.21.15.103:0/1971339815 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6f740377c0 0x7f6f74039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.682+0000 7f6f8ffff700 1 --2- 172.21.15.103:0/1971339815 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6f740377c0 0x7f6f74039cd0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.686+0000 7f6f8d7fa700 1 -- 172.21.15.103:0/1971339815 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6f84009c80 con 0x7f6f90132f50 2023-07-02T08:59:42.723 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.717+0000 7f83352f2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:42.724 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.723+0000 7f83352f2700 1 Processor -- start 2023-07-02T08:59:42.724 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.723+0000 7f83352f2700 1 -- start start 2023-07-02T08:59:42.725 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f83352f2700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8330133360 0x7f833009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.725 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f83352f2700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833009e260 con 0x7f8330132f50 2023-07-02T08:59:42.725 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f83352f2700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83300a1640 con 0x7f8330133360 2023-07-02T08:59:42.726 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832e7fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8330133360 0x7f833009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.726 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832e7fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8330133360 0x7f833009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55304/0 (socket says 172.21.15.103:55304) 2023-07-02T08:59:42.726 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832effd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f8330132f50 0x7f833009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43156/0 (socket says 172.21.15.103:43156) 2023-07-02T08:59:42.726 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832e7fc700 1 -- 172.21.15.103:0/1799810551 learned_addr learned my addr 172.21.15.103:0/1799810551 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:42.726 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832e7fc700 1 -- 172.21.15.103:0/1799810551 >> v1:172.21.15.103:6789/0 conn(0x7f8330132f50 legacy=0x7f833009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.726 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832e7fc700 1 -- 172.21.15.103:0/1799810551 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8330132a20 con 0x7f8330133360 2023-07-02T08:59:42.726 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832e7fc700 1 --2- 172.21.15.103:0/1799810551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330133360 0x7f833009cc90 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56ac66a74689127 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832dffb700 1 -- 172.21.15.103:0/1799810551 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8324009d90 con 0x7f8330133360 2023-07-02T08:59:42.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832dffb700 1 -- 172.21.15.103:0/1799810551 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f832400e230 con 0x7f8330133360 2023-07-02T08:59:42.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.724+0000 7f832dffb700 1 -- 172.21.15.103:0/1799810551 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f832400e4d0 con 0x7f8330133360 2023-07-02T08:59:42.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.725+0000 7f83352f2700 1 -- 172.21.15.103:0/1799810551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330133360 msgr2=0x7f833009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.725+0000 7f83352f2700 1 --2- 172.21.15.103:0/1799810551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330133360 0x7f833009cc90 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.726+0000 7f83352f2700 1 -- 172.21.15.103:0/1799810551 shutdown_connections 2023-07-02T08:59:42.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.726+0000 7f83352f2700 1 --2- 172.21.15.103:0/1799810551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330133360 0x7f833009cc90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.726+0000 7f83352f2700 1 -- 172.21.15.103:0/1799810551 >> 172.21.15.103:0/1799810551 conn(0x7f83300a2080 msgr2=0x7f83300a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:42.729 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.726+0000 7f83352f2700 1 -- 172.21.15.103:0/1799810551 shutdown_connections 2023-07-02T08:59:42.729 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.726+0000 7f83352f2700 1 -- 172.21.15.103:0/1799810551 wait complete. 2023-07-02T08:59:42.729 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.726+0000 7f83352f2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:42.729 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.728+0000 7f83352f2700 1 Processor -- start 2023-07-02T08:59:42.730 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.728+0000 7f83352f2700 1 -- start start 2023-07-02T08:59:42.730 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.729+0000 7f83352f2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330132f50 0x7f833012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.730 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.729+0000 7f83352f2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833006d260 con 0x7f8330132f50 2023-07-02T08:59:42.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.729+0000 7f832effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330132f50 0x7f833012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.729+0000 7f832effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330132f50 0x7f833012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55312/0 (socket says 172.21.15.103:55312) 2023-07-02T08:59:42.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.729+0000 7f832effd700 1 -- 172.21.15.103:0/2124157739 learned_addr learned my addr 172.21.15.103:0/2124157739 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:42.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.729+0000 7f832effd700 1 -- 172.21.15.103:0/2124157739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833012bb90 con 0x7f8330132f50 2023-07-02T08:59:42.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.729+0000 7f832effd700 1 --2- 172.21.15.103:0/2124157739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330132f50 0x7f833012bea0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.730+0000 7f83177fe700 1 -- 172.21.15.103:0/2124157739 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f83200059a0 con 0x7f8330132f50 2023-07-02T08:59:42.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.730+0000 7f83177fe700 1 -- 172.21.15.103:0/2124157739 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8320005b00 con 0x7f8330132f50 2023-07-02T08:59:42.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.730+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833006f730 con 0x7f8330132f50 2023-07-02T08:59:42.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.730+0000 7f83177fe700 1 -- 172.21.15.103:0/2124157739 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8320005da0 con 0x7f8330132f50 2023-07-02T08:59:42.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.730+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833006f490 con 0x7f8330132f50 2023-07-02T08:59:42.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.731+0000 7f83177fe700 1 -- 172.21.15.103:0/2124157739 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f83200070b0 con 0x7f8330132f50 2023-07-02T08:59:42.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.731+0000 7f83177fe700 1 --2- 172.21.15.103:0/2124157739 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f831c0377c0 0x7f831c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.731+0000 7f83177fe700 1 -- 172.21.15.103:0/2124157739 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f83200469e0 con 0x7f8330132f50 2023-07-02T08:59:42.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.731+0000 7f832e7fc700 1 --2- 172.21.15.103:0/2124157739 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f831c0377c0 0x7f831c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.731+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833006f490 con 0x7f8330132f50 2023-07-02T08:59:42.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.732+0000 7f832e7fc700 1 --2- 172.21.15.103:0/2124157739 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f831c0377c0 0x7f831c039cd0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.736+0000 7f83177fe700 1 -- 172.21.15.103:0/2124157739 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f833006f490 con 0x7f8330132f50 2023-07-02T08:59:42.749 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:42.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.753+0000 7f36da47a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.759+0000 7f36da47a700 1 Processor -- start 2023-07-02T08:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.759+0000 7f36da47a700 1 -- start start 2023-07-02T08:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.759+0000 7f36da47a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36d409e3c0 0x7f36d4137c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.759+0000 7f36da47a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d409f2e0 con 0x7f36d409d8a0 2023-07-02T08:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.759+0000 7f36da47a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d4099090 con 0x7f36d409e3c0 2023-07-02T08:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d3fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f36d409d8a0 0x7f36d409dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43164/0 (socket says 172.21.15.103:43164) 2023-07-02T08:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d3fff700 1 -- 172.21.15.103:0/96714748 learned_addr learned my addr 172.21.15.103:0/96714748 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d37fe700 1 --2- 172.21.15.103:0/96714748 >> v2:172.21.15.103:3300/0 conn(0x7f36d409e3c0 0x7f36d4137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d37fe700 1 -- 172.21.15.103:0/96714748 >> v1:172.21.15.103:6789/0 conn(0x7f36d409d8a0 legacy=0x7f36d409dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d37fe700 1 -- 172.21.15.103:0/96714748 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d409a000 con 0x7f36d409e3c0 2023-07-02T08:59:42.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d37fe700 1 --2- 172.21.15.103:0/96714748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409e3c0 0x7f36d4137c90 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff934ecc4210325 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d2ffd700 1 -- 172.21.15.103:0/96714748 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36c8009cd0 con 0x7f36d409e3c0 2023-07-02T08:59:42.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d2ffd700 1 -- 172.21.15.103:0/96714748 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f36c8009e30 con 0x7f36d409e3c0 2023-07-02T08:59:42.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.760+0000 7f36d2ffd700 1 -- 172.21.15.103:0/96714748 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36c800e360 con 0x7f36d409e3c0 2023-07-02T08:59:42.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.761+0000 7f36da47a700 1 -- 172.21.15.103:0/96714748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409e3c0 msgr2=0x7f36d4137c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.761+0000 7f36da47a700 1 --2- 172.21.15.103:0/96714748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409e3c0 0x7f36d4137c90 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.761+0000 7f36da47a700 1 -- 172.21.15.103:0/96714748 shutdown_connections 2023-07-02T08:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.761+0000 7f36da47a700 1 --2- 172.21.15.103:0/96714748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409e3c0 0x7f36d4137c90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.761+0000 7f36da47a700 1 -- 172.21.15.103:0/96714748 >> 172.21.15.103:0/96714748 conn(0x7f36d40a3280 msgr2=0x7f36d40a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.761+0000 7f36da47a700 1 -- 172.21.15.103:0/96714748 shutdown_connections 2023-07-02T08:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.761+0000 7f36da47a700 1 -- 172.21.15.103:0/96714748 wait complete. 2023-07-02T08:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.762+0000 7f36da47a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:42.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.763+0000 7f36da47a700 1 Processor -- start 2023-07-02T08:59:42.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.763+0000 7f36da47a700 1 -- start start 2023-07-02T08:59:42.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.764+0000 7f36da47a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409d8a0 0x7f36d41a7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.764+0000 7f36da47a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41aae00 con 0x7f36d409d8a0 2023-07-02T08:59:42.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.764+0000 7f36d3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409d8a0 0x7f36d41a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.764+0000 7f36d3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409d8a0 0x7f36d41a7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55334/0 (socket says 172.21.15.103:55334) 2023-07-02T08:59:42.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.764+0000 7f36d3fff700 1 -- 172.21.15.103:0/332636897 learned_addr learned my addr 172.21.15.103:0/332636897 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:42.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.765+0000 7f36d3fff700 1 -- 172.21.15.103:0/332636897 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d41bb3a0 con 0x7f36d409d8a0 2023-07-02T08:59:42.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.765+0000 7f36d3fff700 1 --2- 172.21.15.103:0/332636897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409d8a0 0x7f36d41a7850 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.765+0000 7f36d0ff9700 1 -- 172.21.15.103:0/332636897 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36c4005a60 con 0x7f36d409d8a0 2023-07-02T08:59:42.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.765+0000 7f36d0ff9700 1 -- 172.21.15.103:0/332636897 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f36c4005bc0 con 0x7f36d409d8a0 2023-07-02T08:59:42.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.765+0000 7f36d0ff9700 1 -- 172.21.15.103:0/332636897 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36c4005e60 con 0x7f36d409d8a0 2023-07-02T08:59:42.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.765+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d41bb570 con 0x7f36d409d8a0 2023-07-02T08:59:42.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.765+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d41a9690 con 0x7f36d409d8a0 2023-07-02T08:59:42.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.766+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d41a9260 con 0x7f36d409d8a0 2023-07-02T08:59:42.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.767+0000 7f36d0ff9700 1 -- 172.21.15.103:0/332636897 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f36c400c7b0 con 0x7f36d409d8a0 2023-07-02T08:59:42.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.767+0000 7f36d0ff9700 1 --2- 172.21.15.103:0/332636897 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36c00377c0 0x7f36c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.767+0000 7f36d0ff9700 1 -- 172.21.15.103:0/332636897 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f36c4046e30 con 0x7f36d409d8a0 2023-07-02T08:59:42.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.767+0000 7f36d37fe700 1 --2- 172.21.15.103:0/332636897 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36c00377c0 0x7f36c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.768+0000 7f36d37fe700 1 --2- 172.21.15.103:0/332636897 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36c00377c0 0x7f36c0039cd0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.771+0000 7f36d0ff9700 1 -- 172.21.15.103:0/332636897 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f36c400d330 con 0x7f36d409d8a0 2023-07-02T08:59:42.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.877+0000 7f4ce5a7a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:42.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.882+0000 7f4ce5a7a700 1 Processor -- start 2023-07-02T08:59:42.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.882+0000 7f4ce5a7a700 1 -- start start 2023-07-02T08:59:42.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.883+0000 7f4ce5a7a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ce0132f50 0x7f4ce0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.883+0000 7f4ce5a7a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce009e260 con 0x7f4ce009c580 2023-07-02T08:59:42.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.883+0000 7f4ce5a7a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce00a1640 con 0x7f4ce0132f50 2023-07-02T08:59:42.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.883+0000 7f4cdf7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ce0132f50 0x7f4ce0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.883+0000 7f4cdeffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4ce009c580 0x7f4ce009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43166/0 (socket says 172.21.15.103:43166) 2023-07-02T08:59:42.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.883+0000 7f4cdeffd700 1 -- 172.21.15.103:0/4291975736 learned_addr learned my addr 172.21.15.103:0/4291975736 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:42.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.883+0000 7f4cdf7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ce0132f50 0x7f4ce0133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55350/0 (socket says 172.21.15.103:55350) 2023-07-02T08:59:42.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.884+0000 7f4cdf7fe700 1 -- 172.21.15.103:0/4291975736 >> v1:172.21.15.103:6789/0 conn(0x7f4ce009c580 legacy=0x7f4ce009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.884+0000 7f4cde7fc700 1 -- 172.21.15.103:0/4291975736 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802685378 0 0) 0x7f4ce009e260 con 0x7f4ce009c580 2023-07-02T08:59:42.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.884+0000 7f4cdf7fe700 1 -- 172.21.15.103:0/4291975736 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce0132a20 con 0x7f4ce0132f50 2023-07-02T08:59:42.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.884+0000 7f4cdf7fe700 1 --2- 172.21.15.103:0/4291975736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce0132f50 0x7f4ce0133360 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3780706ec813fc server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.884+0000 7f4cde7fc700 1 -- 172.21.15.103:0/4291975736 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4cd000e040 con 0x7f4ce0132f50 2023-07-02T08:59:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.884+0000 7f4cde7fc700 1 -- 172.21.15.103:0/4291975736 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4cd0007670 con 0x7f4ce0132f50 2023-07-02T08:59:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.884+0000 7f4cde7fc700 1 -- 172.21.15.103:0/4291975736 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4cd0007910 con 0x7f4ce0132f50 2023-07-02T08:59:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.885+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/4291975736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce0132f50 msgr2=0x7f4ce0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.885+0000 7f4ce5a7a700 1 --2- 172.21.15.103:0/4291975736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce0132f50 0x7f4ce0133360 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.885+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/4291975736 shutdown_connections 2023-07-02T08:59:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.885+0000 7f4ce5a7a700 1 --2- 172.21.15.103:0/4291975736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce0132f50 0x7f4ce0133360 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.885+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/4291975736 >> 172.21.15.103:0/4291975736 conn(0x7f4ce00a2080 msgr2=0x7f4ce00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.885+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/4291975736 shutdown_connections 2023-07-02T08:59:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.885+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/4291975736 wait complete. 2023-07-02T08:59:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.885+0000 7f4ce5a7a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:42.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.886+0000 7f4ce5a7a700 1 Processor -- start 2023-07-02T08:59:42.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.886+0000 7f4ce5a7a700 1 -- start start 2023-07-02T08:59:42.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.887+0000 7f4ce5a7a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce009c580 0x7f4ce01baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.887+0000 7f4ce5a7a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce01ba610 con 0x7f4ce009c580 2023-07-02T08:59:42.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.887+0000 7f4cdf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce009c580 0x7f4ce01baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.887+0000 7f4cdf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce009c580 0x7f4ce01baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55362/0 (socket says 172.21.15.103:55362) 2023-07-02T08:59:42.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.887+0000 7f4cdf7fe700 1 -- 172.21.15.103:0/1688525605 learned_addr learned my addr 172.21.15.103:0/1688525605 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:42.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.887+0000 7f4cdf7fe700 1 -- 172.21.15.103:0/1688525605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce006dfc0 con 0x7f4ce009c580 2023-07-02T08:59:42.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.887+0000 7f4cdf7fe700 1 --2- 172.21.15.103:0/1688525605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce009c580 0x7f4ce01baa60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.888+0000 7f4cc7fff700 1 -- 172.21.15.103:0/1688525605 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4cd000e040 con 0x7f4ce009c580 2023-07-02T08:59:42.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.888+0000 7f4cc7fff700 1 -- 172.21.15.103:0/1688525605 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4cd0013960 con 0x7f4ce009c580 2023-07-02T08:59:42.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.888+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce01ba270 con 0x7f4ce009c580 2023-07-02T08:59:42.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.888+0000 7f4cc7fff700 1 -- 172.21.15.103:0/1688525605 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4cd0013c00 con 0x7f4ce009c580 2023-07-02T08:59:42.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.888+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce006dba0 con 0x7f4ce009c580 2023-07-02T08:59:42.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.889+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce01ba460 con 0x7f4ce009c580 2023-07-02T08:59:42.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.889+0000 7f4cc7fff700 1 -- 172.21.15.103:0/1688525605 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4cd0011040 con 0x7f4ce009c580 2023-07-02T08:59:42.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.889+0000 7f4cc7fff700 1 --2- 172.21.15.103:0/1688525605 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4ccc0377c0 0x7f4ccc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:42.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.889+0000 7f4cc7fff700 1 -- 172.21.15.103:0/1688525605 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4cd0048380 con 0x7f4ce009c580 2023-07-02T08:59:42.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.890+0000 7f4cdeffd700 1 --2- 172.21.15.103:0/1688525605 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4ccc0377c0 0x7f4ccc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:42.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.890+0000 7f4cdeffd700 1 --2- 172.21.15.103:0/1688525605 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4ccc0377c0 0x7f4ccc039cd0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:42.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.894+0000 7f4cc7fff700 1 -- 172.21.15.103:0/1688525605 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f4cd0010e60 con 0x7f4ce009c580 2023-07-02T08:59:42.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.978+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f36d4061ea0 con 0x7f36d409d8a0 2023-07-02T08:59:42.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.979+0000 7f36d0ff9700 1 -- 172.21.15.103:0/332636897 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f36c400ee40 con 0x7f36d409d8a0 2023-07-02T08:59:42.980 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:42.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36c00377c0 msgr2=0x7f36c0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 --2- 172.21.15.103:0/332636897 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36c00377c0 0x7f36c0039cd0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409d8a0 msgr2=0x7f36d41a7850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 --2- 172.21.15.103:0/332636897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409d8a0 0x7f36d41a7850 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 shutdown_connections 2023-07-02T08:59:42.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 --2- 172.21.15.103:0/332636897 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36c00377c0 0x7f36c0039cd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 --2- 172.21.15.103:0/332636897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d409d8a0 0x7f36d41a7850 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 >> 172.21.15.103:0/332636897 conn(0x7f36d40a3280 msgr2=0x7f36d406f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:42.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 shutdown_connections 2023-07-02T08:59:42.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.981+0000 7f36da47a700 1 -- 172.21.15.103:0/332636897 wait complete. 2023-07-02T08:59:42.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.986+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6f90061ea0 con 0x7f6f90132f50 2023-07-02T08:59:42.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.986+0000 7f6f8d7fa700 1 -- 172.21.15.103:0/1971339815 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6f840098a0 con 0x7f6f90132f50 2023-07-02T08:59:42.988 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:42.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.988+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6f740377c0 msgr2=0x7f6f74039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.988+0000 7f6f970b5700 1 --2- 172.21.15.103:0/1971339815 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6f740377c0 0x7f6f74039cd0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.988+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90132f50 msgr2=0x7f6f9012c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:42.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.988+0000 7f6f970b5700 1 --2- 172.21.15.103:0/1971339815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90132f50 0x7f6f9012c1a0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.989+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 shutdown_connections 2023-07-02T08:59:42.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.989+0000 7f6f970b5700 1 --2- 172.21.15.103:0/1971339815 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6f740377c0 0x7f6f74039cd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.989+0000 7f6f970b5700 1 --2- 172.21.15.103:0/1971339815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f90132f50 0x7f6f9012c1a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:42.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.989+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 >> 172.21.15.103:0/1971339815 conn(0x7f6f900a2080 msgr2=0x7f6f90130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:42.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.989+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 shutdown_connections 2023-07-02T08:59:42.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:42.989+0000 7f6f970b5700 1 -- 172.21.15.103:0/1971339815 wait complete. 2023-07-02T08:59:43.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:43.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:43.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.019+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f833006f490 con 0x7f8330132f50 2023-07-02T08:59:43.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.020+0000 7f83177fe700 1 -- 172.21.15.103:0/2124157739 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f833006f490 con 0x7f8330132f50 2023-07-02T08:59:43.021 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:43.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f831c0377c0 msgr2=0x7f831c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:43.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 --2- 172.21.15.103:0/2124157739 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f831c0377c0 0x7f831c039cd0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:43.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330132f50 msgr2=0x7f833012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:43.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 --2- 172.21.15.103:0/2124157739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330132f50 0x7f833012bea0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:43.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 shutdown_connections 2023-07-02T08:59:43.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 --2- 172.21.15.103:0/2124157739 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f831c0377c0 0x7f831c039cd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:43.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 --2- 172.21.15.103:0/2124157739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8330132f50 0x7f833012bea0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:43.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 >> 172.21.15.103:0/2124157739 conn(0x7f83300a2080 msgr2=0x7f83301bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:43.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 shutdown_connections 2023-07-02T08:59:43.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.022+0000 7f83352f2700 1 -- 172.21.15.103:0/2124157739 wait complete. 2023-07-02T08:59:43.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.099+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4ce0061ea0 con 0x7f4ce009c580 2023-07-02T08:59:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.099+0000 7f4cc7fff700 1 -- 172.21.15.103:0/1688525605 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4cd00112f0 con 0x7f4ce009c580 2023-07-02T08:59:43.101 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:43.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.101+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4ccc0377c0 msgr2=0x7f4ccc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:43.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.101+0000 7f4ce5a7a700 1 --2- 172.21.15.103:0/1688525605 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4ccc0377c0 0x7f4ccc039cd0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:43.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.101+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce009c580 msgr2=0x7f4ce01baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:43.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.101+0000 7f4ce5a7a700 1 --2- 172.21.15.103:0/1688525605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce009c580 0x7f4ce01baa60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:43.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.102+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 shutdown_connections 2023-07-02T08:59:43.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.102+0000 7f4ce5a7a700 1 --2- 172.21.15.103:0/1688525605 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4ccc0377c0 0x7f4ccc039cd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:43.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.102+0000 7f4ce5a7a700 1 --2- 172.21.15.103:0/1688525605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ce009c580 0x7f4ce01baa60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:43.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.102+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 >> 172.21.15.103:0/1688525605 conn(0x7f4ce00a2080 msgr2=0x7f4ce0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:43.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.102+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 shutdown_connections 2023-07-02T08:59:43.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:43.102+0000 7f4ce5a7a700 1 -- 172.21.15.103:0/1688525605 wait complete. 2023-07-02T08:59:43.115 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:44.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:44.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:44.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:44.116 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:44.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.141+0000 7ff37a25d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:44.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.148+0000 7ff37a25d700 1 Processor -- start 2023-07-02T08:59:44.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.148+0000 7ff37a25d700 1 -- start start 2023-07-02T08:59:44.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.148+0000 7ff37a25d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff374132f50 0x7ff374133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.148+0000 7ff37a25d700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37409e260 con 0x7ff37409c580 2023-07-02T08:59:44.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.148+0000 7ff37a25d700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740a1640 con 0x7ff374132f50 2023-07-02T08:59:44.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.149+0000 7ff3737fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff374132f50 0x7ff374133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.149+0000 7ff3737fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff374132f50 0x7ff374133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36570/0 (socket says 172.21.15.103:36570) 2023-07-02T08:59:44.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.149+0000 7ff3737fe700 1 -- 172.21.15.103:0/3032529463 learned_addr learned my addr 172.21.15.103:0/3032529463 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:44.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.149+0000 7ff372ffd700 1 --1- 172.21.15.103:0/3032529463 >> v1:172.21.15.103:6789/0 conn(0x7ff37409c580 0x7ff37409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55554/0 (socket says 172.21.15.103:55554) 2023-07-02T08:59:44.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.149+0000 7ff3737fe700 1 -- 172.21.15.103:0/3032529463 >> v1:172.21.15.103:6789/0 conn(0x7ff37409c580 legacy=0x7ff37409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.149+0000 7ff3737fe700 1 -- 172.21.15.103:0/3032529463 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff374132a20 con 0x7ff374132f50 2023-07-02T08:59:44.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.149+0000 7ff3737fe700 1 --2- 172.21.15.103:0/3032529463 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff374132f50 0x7ff374133360 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f833f71c582ba6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.151+0000 7ff3727fc700 1 -- 172.21.15.103:0/3032529463 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff364009d90 con 0x7ff374132f50 2023-07-02T08:59:44.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.151+0000 7ff3727fc700 1 -- 172.21.15.103:0/3032529463 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff36400e230 con 0x7ff374132f50 2023-07-02T08:59:44.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.151+0000 7ff3727fc700 1 -- 172.21.15.103:0/3032529463 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff36400e4d0 con 0x7ff374132f50 2023-07-02T08:59:44.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.152+0000 7ff37a25d700 1 -- 172.21.15.103:0/3032529463 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff374132f50 msgr2=0x7ff374133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.152+0000 7ff37a25d700 1 --2- 172.21.15.103:0/3032529463 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff374132f50 0x7ff374133360 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.152+0000 7ff37a25d700 1 -- 172.21.15.103:0/3032529463 shutdown_connections 2023-07-02T08:59:44.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.152+0000 7ff37a25d700 1 --2- 172.21.15.103:0/3032529463 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff374132f50 0x7ff374133360 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.152+0000 7ff37a25d700 1 -- 172.21.15.103:0/3032529463 >> 172.21.15.103:0/3032529463 conn(0x7ff3740a2080 msgr2=0x7ff3740a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:44.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.152+0000 7ff37a25d700 1 -- 172.21.15.103:0/3032529463 shutdown_connections 2023-07-02T08:59:44.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.152+0000 7ff37a25d700 1 -- 172.21.15.103:0/3032529463 wait complete. 2023-07-02T08:59:44.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.152+0000 7ff37a25d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:44.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.154+0000 7ff37a25d700 1 Processor -- start 2023-07-02T08:59:44.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.154+0000 7ff37a25d700 1 -- start start 2023-07-02T08:59:44.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.155+0000 7ff37a25d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37409c580 0x7ff37412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.155+0000 7ff37a25d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37406f9d0 con 0x7ff37409c580 2023-07-02T08:59:44.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.155+0000 7ff3737fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37409c580 0x7ff37412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.155+0000 7ff3737fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37409c580 0x7ff37412b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36584/0 (socket says 172.21.15.103:36584) 2023-07-02T08:59:44.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.155+0000 7ff3737fe700 1 -- 172.21.15.103:0/1335662376 learned_addr learned my addr 172.21.15.103:0/1335662376 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:44.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.155+0000 7ff3737fe700 1 -- 172.21.15.103:0/1335662376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff374133a00 con 0x7ff37409c580 2023-07-02T08:59:44.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.156+0000 7ff3737fe700 1 --2- 172.21.15.103:0/1335662376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37409c580 0x7ff37412b7b0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.156+0000 7ff35bfff700 1 -- 172.21.15.103:0/1335662376 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff364014540 con 0x7ff37409c580 2023-07-02T08:59:44.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.156+0000 7ff35bfff700 1 -- 172.21.15.103:0/1335662376 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff3640146a0 con 0x7ff37409c580 2023-07-02T08:59:44.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.156+0000 7ff35bfff700 1 -- 172.21.15.103:0/1335662376 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff364014940 con 0x7ff37409c580 2023-07-02T08:59:44.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.156+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37406d160 con 0x7ff37409c580 2023-07-02T08:59:44.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.148+0000 7f3b2e74e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:44.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.156+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37406da70 con 0x7ff37409c580 2023-07-02T08:59:44.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.157+0000 7ff35bfff700 1 -- 172.21.15.103:0/1335662376 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ff364014af0 con 0x7ff37409c580 2023-07-02T08:59:44.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.157+0000 7ff35bfff700 1 --2- 172.21.15.103:0/1335662376 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff3600377b0 0x7ff360039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.157+0000 7ff35bfff700 1 -- 172.21.15.103:0/1335662376 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ff364048350 con 0x7ff37409c580 2023-07-02T08:59:44.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.157+0000 7ff372ffd700 1 --2- 172.21.15.103:0/1335662376 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff3600377b0 0x7ff360039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.158+0000 7ff372ffd700 1 --2- 172.21.15.103:0/1335662376 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff3600377b0 0x7ff360039cc0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.158+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff374061ea0 con 0x7ff37409c580 2023-07-02T08:59:44.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.158+0000 7f3b2e74e700 1 Processor -- start 2023-07-02T08:59:44.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.158+0000 7f3b2e74e700 1 -- start start 2023-07-02T08:59:44.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.159+0000 7f3b2e74e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3b2812f2d0 0x7f3b2812f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.159+0000 7f3b2e74e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2812ec50 con 0x7f3b28098460 2023-07-02T08:59:44.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.159+0000 7f3b2e74e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2809a280 con 0x7f3b2812f2d0 2023-07-02T08:59:44.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.160+0000 7f3b27fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3b2812f2d0 0x7f3b2812f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.160+0000 7f3b277fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3b28098460 0x7f3b28098870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55556/0 (socket says 172.21.15.103:55556) 2023-07-02T08:59:44.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.160+0000 7f3b277fe700 1 -- 172.21.15.103:0/3587603190 learned_addr learned my addr 172.21.15.103:0/3587603190 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:44.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.160+0000 7f3b27fff700 1 -- 172.21.15.103:0/3587603190 >> v1:172.21.15.103:6789/0 conn(0x7f3b28098460 legacy=0x7f3b28098870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.160+0000 7f3b27fff700 1 -- 172.21.15.103:0/3587603190 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28099fe0 con 0x7f3b2812f2d0 2023-07-02T08:59:44.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.160+0000 7f3b27fff700 1 --2- 172.21.15.103:0/3587603190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b2812f2d0 0x7f3b2812f6e0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8176a9eb257ff94 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.165 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.164+0000 7f3b26ffd700 1 -- 172.21.15.103:0/3587603190 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3b18009cd0 con 0x7f3b2812f2d0 2023-07-02T08:59:44.165 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.164+0000 7f3b26ffd700 1 -- 172.21.15.103:0/3587603190 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3b18009e30 con 0x7f3b2812f2d0 2023-07-02T08:59:44.165 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.164+0000 7ff35bfff700 1 -- 172.21.15.103:0/1335662376 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ff36400fe40 con 0x7ff37409c580 2023-07-02T08:59:44.166 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.164+0000 7f3b26ffd700 1 -- 172.21.15.103:0/3587603190 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3b1800e360 con 0x7f3b2812f2d0 2023-07-02T08:59:44.166 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.165+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3587603190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b2812f2d0 msgr2=0x7f3b2812f6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.166 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.165+0000 7f3b2e74e700 1 --2- 172.21.15.103:0/3587603190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b2812f2d0 0x7f3b2812f6e0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.166 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.165+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3587603190 shutdown_connections 2023-07-02T08:59:44.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.165+0000 7f3b2e74e700 1 --2- 172.21.15.103:0/3587603190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b2812f2d0 0x7f3b2812f6e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.165+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3587603190 >> 172.21.15.103:0/3587603190 conn(0x7f3b280a1030 msgr2=0x7f3b280a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:44.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.165+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3587603190 shutdown_connections 2023-07-02T08:59:44.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.165+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3587603190 wait complete. 2023-07-02T08:59:44.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.165+0000 7f3b2e74e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:44.168 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.166+0000 7f3b2e74e700 1 Processor -- start 2023-07-02T08:59:44.168 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.167+0000 7f3b2e74e700 1 -- start start 2023-07-02T08:59:44.169 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.167+0000 7f3b2e74e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b28098460 0x7f3b28071db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.169 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.167+0000 7f3b2e74e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b28136610 con 0x7f3b28098460 2023-07-02T08:59:44.169 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.168+0000 7f3b27fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b28098460 0x7f3b28071db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.170 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.168+0000 7f3b27fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b28098460 0x7f3b28071db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36610/0 (socket says 172.21.15.103:36610) 2023-07-02T08:59:44.170 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.168+0000 7f3b27fff700 1 -- 172.21.15.103:0/3450890335 learned_addr learned my addr 172.21.15.103:0/3450890335 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:44.170 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.168+0000 7f3b27fff700 1 -- 172.21.15.103:0/3450890335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28138180 con 0x7f3b28098460 2023-07-02T08:59:44.170 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.168+0000 7f3b27fff700 1 --2- 172.21.15.103:0/3450890335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b28098460 0x7f3b28071db0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.170 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.168+0000 7f3b24ff9700 1 -- 172.21.15.103:0/3450890335 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3b18009c50 con 0x7f3b28098460 2023-07-02T08:59:44.170 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.168+0000 7f3b24ff9700 1 -- 172.21.15.103:0/3450890335 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3b18013610 con 0x7f3b28098460 2023-07-02T08:59:44.170 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.168+0000 7f3b24ff9700 1 -- 172.21.15.103:0/3450890335 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3b180138b0 con 0x7f3b28098460 2023-07-02T08:59:44.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.169+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b28134710 con 0x7f3b28098460 2023-07-02T08:59:44.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.169+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b28138d30 con 0x7f3b28098460 2023-07-02T08:59:44.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.170+0000 7f3b24ff9700 1 -- 172.21.15.103:0/3450890335 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3b18013a60 con 0x7f3b28098460 2023-07-02T08:59:44.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.170+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b28136440 con 0x7f3b28098460 2023-07-02T08:59:44.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.171+0000 7f3b24ff9700 1 --2- 172.21.15.103:0/3450890335 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b080377b0 0x7f3b08039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.171+0000 7f3b24ff9700 1 -- 172.21.15.103:0/3450890335 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3b18047ea0 con 0x7f3b28098460 2023-07-02T08:59:44.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.171+0000 7f3b277fe700 1 --2- 172.21.15.103:0/3450890335 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b080377b0 0x7f3b08039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.172+0000 7f3b277fe700 1 --2- 172.21.15.103:0/3450890335 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b080377b0 0x7f3b08039cc0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.176+0000 7f3b24ff9700 1 -- 172.21.15.103:0/3450890335 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3b18019760 con 0x7f3b28098460 2023-07-02T08:59:44.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.189+0000 7efea4e06700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:44.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.195+0000 7efea4e06700 1 Processor -- start 2023-07-02T08:59:44.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.195+0000 7efea4e06700 1 -- start start 2023-07-02T08:59:44.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efea4e06700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efea0133360 0x7efea009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efea4e06700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea009e260 con 0x7efea0132f50 2023-07-02T08:59:44.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efea4e06700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea00a1640 con 0x7efea0133360 2023-07-02T08:59:44.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efe9dd9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efea0133360 0x7efea009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efe9e59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7efea0132f50 0x7efea009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55558/0 (socket says 172.21.15.103:55558) 2023-07-02T08:59:44.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efe9e59c700 1 -- 172.21.15.103:0/2826391924 learned_addr learned my addr 172.21.15.103:0/2826391924 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:44.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efe9d59a700 1 -- 172.21.15.103:0/2826391924 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985406923 0 0) 0x7efea009e260 con 0x7efea0132f50 2023-07-02T08:59:44.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efe9d59a700 1 -- 172.21.15.103:0/2826391924 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe88003580 con 0x7efea0132f50 2023-07-02T08:59:44.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efe9dd9b700 1 -- 172.21.15.103:0/2826391924 >> v1:172.21.15.103:6789/0 conn(0x7efea0132f50 legacy=0x7efea009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.196+0000 7efe9dd9b700 1 -- 172.21.15.103:0/2826391924 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0132a20 con 0x7efea0133360 2023-07-02T08:59:44.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.197+0000 7efe9dd9b700 1 --2- 172.21.15.103:0/2826391924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0133360 0x7efea009cc90 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c6227916eaa089 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.197+0000 7efe9d59a700 1 -- 172.21.15.103:0/2826391924 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efe94009c80 con 0x7efea0133360 2023-07-02T08:59:44.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.197+0000 7efe9d59a700 1 -- 172.21.15.103:0/2826391924 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efe94009de0 con 0x7efea0133360 2023-07-02T08:59:44.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.197+0000 7efe9d59a700 1 -- 172.21.15.103:0/2826391924 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efe9400e360 con 0x7efea0133360 2023-07-02T08:59:44.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.198+0000 7efea4e06700 1 -- 172.21.15.103:0/2826391924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0133360 msgr2=0x7efea009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.198+0000 7efea4e06700 1 --2- 172.21.15.103:0/2826391924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0133360 0x7efea009cc90 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.198+0000 7efea4e06700 1 -- 172.21.15.103:0/2826391924 shutdown_connections 2023-07-02T08:59:44.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.198+0000 7efea4e06700 1 --2- 172.21.15.103:0/2826391924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0133360 0x7efea009cc90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.198+0000 7efea4e06700 1 -- 172.21.15.103:0/2826391924 >> 172.21.15.103:0/2826391924 conn(0x7efea00a2080 msgr2=0x7efea00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:44.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.198+0000 7efea4e06700 1 -- 172.21.15.103:0/2826391924 shutdown_connections 2023-07-02T08:59:44.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.198+0000 7efea4e06700 1 -- 172.21.15.103:0/2826391924 wait complete. 2023-07-02T08:59:44.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.198+0000 7efea4e06700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:44.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.199+0000 7efea4e06700 1 Processor -- start 2023-07-02T08:59:44.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.199+0000 7efea4e06700 1 -- start start 2023-07-02T08:59:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.200+0000 7efea4e06700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0132f50 0x7efea008cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.200+0000 7efea4e06700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea008a480 con 0x7efea0132f50 2023-07-02T08:59:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.200+0000 7efe9e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0132f50 0x7efea008cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.200+0000 7efe9e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0132f50 0x7efea008cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36624/0 (socket says 172.21.15.103:36624) 2023-07-02T08:59:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.200+0000 7efe9e59c700 1 -- 172.21.15.103:0/433871550 learned_addr learned my addr 172.21.15.103:0/433871550 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.200+0000 7efe9e59c700 1 -- 172.21.15.103:0/433871550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea008bc90 con 0x7efea0132f50 2023-07-02T08:59:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.201+0000 7efe9e59c700 1 --2- 172.21.15.103:0/433871550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0132f50 0x7efea008cc80 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.201+0000 7efe8effd700 1 -- 172.21.15.103:0/433871550 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efe90005a60 con 0x7efea0132f50 2023-07-02T08:59:44.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.201+0000 7efe8effd700 1 -- 172.21.15.103:0/433871550 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efe90005bc0 con 0x7efea0132f50 2023-07-02T08:59:44.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.201+0000 7efe8effd700 1 -- 172.21.15.103:0/433871550 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efe90005e60 con 0x7efea0132f50 2023-07-02T08:59:44.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.201+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea008c970 con 0x7efea0132f50 2023-07-02T08:59:44.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.201+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea008c510 con 0x7efea0132f50 2023-07-02T08:59:44.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.202+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea008e0d0 con 0x7efea0132f50 2023-07-02T08:59:44.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.203+0000 7efe8effd700 1 -- 172.21.15.103:0/433871550 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7efe900070b0 con 0x7efea0132f50 2023-07-02T08:59:44.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.203+0000 7efe8effd700 1 --2- 172.21.15.103:0/433871550 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efe800377c0 0x7efe80039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.203+0000 7efe8effd700 1 -- 172.21.15.103:0/433871550 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7efe90046db0 con 0x7efea0132f50 2023-07-02T08:59:44.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.205+0000 7efe9dd9b700 1 --2- 172.21.15.103:0/433871550 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efe800377c0 0x7efe80039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.205+0000 7efe9dd9b700 1 --2- 172.21.15.103:0/433871550 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efe800377c0 0x7efe80039cd0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.212+0000 7efe8effd700 1 -- 172.21.15.103:0/433871550 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7efe9000d980 con 0x7efea0132f50 2023-07-02T08:59:44.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.253+0000 7f82dc124700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:44.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.258+0000 7f82dc124700 1 Processor -- start 2023-07-02T08:59:44.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.258+0000 7f82dc124700 1 -- start start 2023-07-02T08:59:44.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.259+0000 7f82dc124700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f82d4133360 0x7f82d409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.259+0000 7f82dc124700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d409e260 con 0x7f82d4132f50 2023-07-02T08:59:44.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.259+0000 7f82dc124700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d40a1640 con 0x7f82d4133360 2023-07-02T08:59:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.259+0000 7f82d96bf700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f82d4133360 0x7f82d409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.259+0000 7f82d9ec0700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f82d4132f50 0x7f82d409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55568/0 (socket says 172.21.15.103:55568) 2023-07-02T08:59:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.259+0000 7f82d9ec0700 1 -- 172.21.15.103:0/4211812234 learned_addr learned my addr 172.21.15.103:0/4211812234 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.259+0000 7f82d96bf700 1 -- 172.21.15.103:0/4211812234 >> v1:172.21.15.103:6789/0 conn(0x7f82d4132f50 legacy=0x7f82d409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.259+0000 7f82d96bf700 1 -- 172.21.15.103:0/4211812234 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d4132a20 con 0x7f82d4133360 2023-07-02T08:59:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.260+0000 7f82d96bf700 1 --2- 172.21.15.103:0/4211812234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4133360 0x7f82d409cc90 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e62c1ae91a94ce1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.260+0000 7f82d8ebe700 1 -- 172.21.15.103:0/4211812234 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f82c4009cd0 con 0x7f82d4133360 2023-07-02T08:59:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.260+0000 7f82d8ebe700 1 -- 172.21.15.103:0/4211812234 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f82c4009e30 con 0x7f82d4133360 2023-07-02T08:59:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.260+0000 7f82d8ebe700 1 -- 172.21.15.103:0/4211812234 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f82c400e360 con 0x7f82d4133360 2023-07-02T08:59:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.260+0000 7f82dc124700 1 -- 172.21.15.103:0/4211812234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4133360 msgr2=0x7f82d409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.260+0000 7f82dc124700 1 --2- 172.21.15.103:0/4211812234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4133360 0x7f82d409cc90 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.261+0000 7f82dc124700 1 -- 172.21.15.103:0/4211812234 shutdown_connections 2023-07-02T08:59:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.261+0000 7f82dc124700 1 --2- 172.21.15.103:0/4211812234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4133360 0x7f82d409cc90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.261+0000 7f82dc124700 1 -- 172.21.15.103:0/4211812234 >> 172.21.15.103:0/4211812234 conn(0x7f82d40a2080 msgr2=0x7f82d40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.261+0000 7f82dc124700 1 -- 172.21.15.103:0/4211812234 shutdown_connections 2023-07-02T08:59:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.261+0000 7f82dc124700 1 -- 172.21.15.103:0/4211812234 wait complete. 2023-07-02T08:59:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.261+0000 7f82dc124700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.262+0000 7f82dc124700 1 Processor -- start 2023-07-02T08:59:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.262+0000 7f82dc124700 1 -- start start 2023-07-02T08:59:44.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.262+0000 7f82dc124700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4132f50 0x7f82d412bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.262+0000 7f82dc124700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d406d260 con 0x7f82d4132f50 2023-07-02T08:59:44.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82d9ec0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4132f50 0x7f82d412bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82d9ec0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4132f50 0x7f82d412bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36644/0 (socket says 172.21.15.103:36644) 2023-07-02T08:59:44.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82d9ec0700 1 -- 172.21.15.103:0/1733696683 learned_addr learned my addr 172.21.15.103:0/1733696683 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:44.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82d9ec0700 1 -- 172.21.15.103:0/1733696683 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d412bb90 con 0x7f82d4132f50 2023-07-02T08:59:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82d9ec0700 1 --2- 172.21.15.103:0/1733696683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4132f50 0x7f82d412bea0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82ca7fc700 1 -- 172.21.15.103:0/1733696683 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f82d0005a60 con 0x7f82d4132f50 2023-07-02T08:59:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82ca7fc700 1 -- 172.21.15.103:0/1733696683 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f82d0005bc0 con 0x7f82d4132f50 2023-07-02T08:59:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d406f730 con 0x7f82d4132f50 2023-07-02T08:59:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82ca7fc700 1 -- 172.21.15.103:0/1733696683 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f82d0005e60 con 0x7f82d4132f50 2023-07-02T08:59:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.263+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d406f490 con 0x7f82d4132f50 2023-07-02T08:59:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.264+0000 7f82ca7fc700 1 -- 172.21.15.103:0/1733696683 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f82d00070b0 con 0x7f82d4132f50 2023-07-02T08:59:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.265+0000 7f82ca7fc700 1 --2- 172.21.15.103:0/1733696683 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82b8037810 0x7f82b8039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.265+0000 7f82ca7fc700 1 -- 172.21.15.103:0/1733696683 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f82d0046ab0 con 0x7f82d4132f50 2023-07-02T08:59:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.265+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc0050b0 con 0x7f82d4132f50 2023-07-02T08:59:44.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.272+0000 7f82d96bf700 1 --2- 172.21.15.103:0/1733696683 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82b8037810 0x7f82b8039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:44.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.272+0000 7f82d96bf700 1 --2- 172.21.15.103:0/1733696683 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82b8037810 0x7f82b8039d20 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:44.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.274+0000 7f82ca7fc700 1 -- 172.21.15.103:0/1733696683 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f82d000c750 con 0x7f82d4132f50 2023-07-02T08:59:44.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.370+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff3740725b0 con 0x7ff37409c580 2023-07-02T08:59:44.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.370+0000 7ff35bfff700 1 -- 172.21.15.103:0/1335662376 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ff364010e60 con 0x7ff37409c580 2023-07-02T08:59:44.371 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:44.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.373+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff3600377b0 msgr2=0x7ff360039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.373+0000 7ff37a25d700 1 --2- 172.21.15.103:0/1335662376 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff3600377b0 0x7ff360039cc0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.373+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37409c580 msgr2=0x7ff37412b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.373+0000 7ff37a25d700 1 --2- 172.21.15.103:0/1335662376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37409c580 0x7ff37412b7b0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.374+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 shutdown_connections 2023-07-02T08:59:44.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.374+0000 7ff37a25d700 1 --2- 172.21.15.103:0/1335662376 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff3600377b0 0x7ff360039cc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.374+0000 7ff37a25d700 1 --2- 172.21.15.103:0/1335662376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff37409c580 0x7ff37412b7b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.374+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 >> 172.21.15.103:0/1335662376 conn(0x7ff3740a2080 msgr2=0x7ff3741babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.374+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 shutdown_connections 2023-07-02T08:59:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.374+0000 7ff37a25d700 1 -- 172.21.15.103:0/1335662376 wait complete. 2023-07-02T08:59:44.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:44.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.402+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3b28061ea0 con 0x7f3b28098460 2023-07-02T08:59:44.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.402+0000 7f3b24ff9700 1 -- 172.21.15.103:0/3450890335 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3b180191a0 con 0x7f3b28098460 2023-07-02T08:59:44.404 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:44.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b080377b0 msgr2=0x7f3b08039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 --2- 172.21.15.103:0/3450890335 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b080377b0 0x7f3b08039cc0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b28098460 msgr2=0x7f3b28071db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 --2- 172.21.15.103:0/3450890335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b28098460 0x7f3b28071db0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 shutdown_connections 2023-07-02T08:59:44.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 --2- 172.21.15.103:0/3450890335 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b080377b0 0x7f3b08039cc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 --2- 172.21.15.103:0/3450890335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b28098460 0x7f3b28071db0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 >> 172.21.15.103:0/3450890335 conn(0x7f3b280a1030 msgr2=0x7f3b2812ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:44.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 shutdown_connections 2023-07-02T08:59:44.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.404+0000 7f3b2e74e700 1 -- 172.21.15.103:0/3450890335 wait complete. 2023-07-02T08:59:44.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.416+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efea00a1920 con 0x7efea0132f50 2023-07-02T08:59:44.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.416+0000 7efe8effd700 1 -- 172.21.15.103:0/433871550 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7efe9000d150 con 0x7efea0132f50 2023-07-02T08:59:44.418 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:44.419 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:44.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efe800377c0 msgr2=0x7efe80039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 --2- 172.21.15.103:0/433871550 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efe800377c0 0x7efe80039cd0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0132f50 msgr2=0x7efea008cc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 --2- 172.21.15.103:0/433871550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0132f50 0x7efea008cc80 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 shutdown_connections 2023-07-02T08:59:44.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 --2- 172.21.15.103:0/433871550 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efe800377c0 0x7efe80039cd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 --2- 172.21.15.103:0/433871550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea0132f50 0x7efea008cc80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 >> 172.21.15.103:0/433871550 conn(0x7efea00a2080 msgr2=0x7efea01bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:44.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 shutdown_connections 2023-07-02T08:59:44.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.419+0000 7efea4e06700 1 -- 172.21.15.103:0/433871550 wait complete. 2023-07-02T08:59:44.433 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:44.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.472+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f82bc005360 con 0x7f82d4132f50 2023-07-02T08:59:44.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.473+0000 7f82ca7fc700 1 -- 172.21.15.103:0/1733696683 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f82d000a4a0 con 0x7f82d4132f50 2023-07-02T08:59:44.474 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:44.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82b8037810 msgr2=0x7f82b8039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 --2- 172.21.15.103:0/1733696683 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82b8037810 0x7f82b8039d20 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4132f50 msgr2=0x7f82d412bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:44.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 --2- 172.21.15.103:0/1733696683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4132f50 0x7f82d412bea0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 shutdown_connections 2023-07-02T08:59:44.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 --2- 172.21.15.103:0/1733696683 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82b8037810 0x7f82b8039d20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 --2- 172.21.15.103:0/1733696683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82d4132f50 0x7f82d412bea0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:44.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 >> 172.21.15.103:0/1733696683 conn(0x7f82d40a2080 msgr2=0x7f82d41bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:44.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 shutdown_connections 2023-07-02T08:59:44.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:44.475+0000 7f82dc124700 1 -- 172.21.15.103:0/1733696683 wait complete. 2023-07-02T08:59:44.488 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:45.395 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:45.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:45.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:45.489 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.530+0000 7fc8464fd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.538+0000 7fc8464fd700 1 Processor -- start 2023-07-02T08:59:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.538+0000 7fc8464fd700 1 -- start start 2023-07-02T08:59:45.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.539+0000 7fc8464fd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc840132f50 0x7fc840133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.539+0000 7fc8464fd700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84009e260 con 0x7fc84009c580 2023-07-02T08:59:45.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.539+0000 7fc8464fd700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8400a1640 con 0x7fc840132f50 2023-07-02T08:59:45.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.539+0000 7fc83ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc840132f50 0x7fc840133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.539+0000 7fc83f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc84009c580 0x7fc84009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55578/0 (socket says 172.21.15.103:55578) 2023-07-02T08:59:45.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.539+0000 7fc83f7fe700 1 -- 172.21.15.103:0/1442910838 learned_addr learned my addr 172.21.15.103:0/1442910838 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:45.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.539+0000 7fc83ffff700 1 -- 172.21.15.103:0/1442910838 >> v1:172.21.15.103:6789/0 conn(0x7fc84009c580 legacy=0x7fc84009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.539+0000 7fc83ffff700 1 -- 172.21.15.103:0/1442910838 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc840132a20 con 0x7fc840132f50 2023-07-02T08:59:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.540+0000 7fc83ffff700 1 --2- 172.21.15.103:0/1442910838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc840132f50 0x7fc840133360 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd9445a02c88465 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.540+0000 7fc83effd700 1 -- 172.21.15.103:0/1442910838 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc830009cd0 con 0x7fc840132f50 2023-07-02T08:59:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.540+0000 7fc83effd700 1 -- 172.21.15.103:0/1442910838 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc830009e30 con 0x7fc840132f50 2023-07-02T08:59:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.540+0000 7fc83effd700 1 -- 172.21.15.103:0/1442910838 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc83000e360 con 0x7fc840132f50 2023-07-02T08:59:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.541+0000 7fc8464fd700 1 -- 172.21.15.103:0/1442910838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc840132f50 msgr2=0x7fc840133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.541+0000 7fc8464fd700 1 --2- 172.21.15.103:0/1442910838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc840132f50 0x7fc840133360 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.541+0000 7fc8464fd700 1 -- 172.21.15.103:0/1442910838 shutdown_connections 2023-07-02T08:59:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.541+0000 7fc8464fd700 1 --2- 172.21.15.103:0/1442910838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc840132f50 0x7fc840133360 crc :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.541+0000 7fc8464fd700 1 -- 172.21.15.103:0/1442910838 >> 172.21.15.103:0/1442910838 conn(0x7fc8400a2080 msgr2=0x7fc8400a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.541+0000 7fc8464fd700 1 -- 172.21.15.103:0/1442910838 shutdown_connections 2023-07-02T08:59:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.541+0000 7fc8464fd700 1 -- 172.21.15.103:0/1442910838 wait complete. 2023-07-02T08:59:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.541+0000 7fc8464fd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.543+0000 7fc8464fd700 1 Processor -- start 2023-07-02T08:59:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.543+0000 7fc8464fd700 1 -- start start 2023-07-02T08:59:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.544+0000 7fc8464fd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc84009c580 0x7fc84012c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.544+0000 7fc8464fd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84006e220 con 0x7fc84009c580 2023-07-02T08:59:45.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.544+0000 7fc83ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc84009c580 0x7fc84012c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.544+0000 7fc83ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc84009c580 0x7fc84012c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36656/0 (socket says 172.21.15.103:36656) 2023-07-02T08:59:45.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.544+0000 7fc83ffff700 1 -- 172.21.15.103:0/288803802 learned_addr learned my addr 172.21.15.103:0/288803802 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.545+0000 7fc83ffff700 1 -- 172.21.15.103:0/288803802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84006d400 con 0x7fc84009c580 2023-07-02T08:59:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.545+0000 7fc83ffff700 1 --2- 172.21.15.103:0/288803802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc84009c580 0x7fc84012c0f0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.545+0000 7fc83cff9700 1 -- 172.21.15.103:0/288803802 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc830013530 con 0x7fc84009c580 2023-07-02T08:59:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.545+0000 7fc83cff9700 1 -- 172.21.15.103:0/288803802 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc830013690 con 0x7fc84009c580 2023-07-02T08:59:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.545+0000 7fc83cff9700 1 -- 172.21.15.103:0/288803802 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc830013930 con 0x7fc84009c580 2023-07-02T08:59:45.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.546+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84012bd70 con 0x7fc84009c580 2023-07-02T08:59:45.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.546+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84006ea80 con 0x7fc84009c580 2023-07-02T08:59:45.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.547+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84012bf60 con 0x7fc84009c580 2023-07-02T08:59:45.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.548+0000 7fc83cff9700 1 -- 172.21.15.103:0/288803802 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fc830013ae0 con 0x7fc84009c580 2023-07-02T08:59:45.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.548+0000 7fc83cff9700 1 --2- 172.21.15.103:0/288803802 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc82c0377c0 0x7fc82c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.548+0000 7fc83cff9700 1 -- 172.21.15.103:0/288803802 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc830018040 con 0x7fc84009c580 2023-07-02T08:59:45.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.548+0000 7fc83f7fe700 1 --2- 172.21.15.103:0/288803802 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc82c0377c0 0x7fc82c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.549+0000 7fc83f7fe700 1 --2- 172.21.15.103:0/288803802 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc82c0377c0 0x7fc82c039cd0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.557+0000 7fc83cff9700 1 -- 172.21.15.103:0/288803802 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc830018e00 con 0x7fc84009c580 2023-07-02T08:59:45.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.577+0000 7f246adc8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:45.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.582+0000 7f246adc8700 1 Processor -- start 2023-07-02T08:59:45.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.582+0000 7f246adc8700 1 -- start start 2023-07-02T08:59:45.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f246adc8700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2464133360 0x7f246409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f246adc8700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246409e260 con 0x7f2464132f50 2023-07-02T08:59:45.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f246adc8700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24640a1640 con 0x7f2464133360 2023-07-02T08:59:45.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f2463fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2464133360 0x7f246409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f2468b64700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2464132f50 0x7f246409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55582/0 (socket says 172.21.15.103:55582) 2023-07-02T08:59:45.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f2468b64700 1 -- 172.21.15.103:0/3654356322 learned_addr learned my addr 172.21.15.103:0/3654356322 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:45.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f2463fff700 1 -- 172.21.15.103:0/3654356322 >> v1:172.21.15.103:6789/0 conn(0x7f2464132f50 legacy=0x7f246409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f2463fff700 1 -- 172.21.15.103:0/3654356322 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2464132a20 con 0x7f2464133360 2023-07-02T08:59:45.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f24637fe700 1 -- 172.21.15.103:0/3654356322 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273139065 0 0) 0x7f246409e260 con 0x7f2464132f50 2023-07-02T08:59:45.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f2463fff700 1 --2- 172.21.15.103:0/3654356322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464133360 0x7f246409cc90 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f17b2549d1b6d1e server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.583+0000 7f24637fe700 1 -- 172.21.15.103:0/3654356322 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2450009cd0 con 0x7f2464133360 2023-07-02T08:59:45.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.584+0000 7f24637fe700 1 -- 172.21.15.103:0/3654356322 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2450009e30 con 0x7f2464133360 2023-07-02T08:59:45.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.584+0000 7f24637fe700 1 -- 172.21.15.103:0/3654356322 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f245000e360 con 0x7f2464133360 2023-07-02T08:59:45.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.584+0000 7f246adc8700 1 -- 172.21.15.103:0/3654356322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464133360 msgr2=0x7f246409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.584+0000 7f246adc8700 1 --2- 172.21.15.103:0/3654356322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464133360 0x7f246409cc90 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.584+0000 7f246adc8700 1 -- 172.21.15.103:0/3654356322 shutdown_connections 2023-07-02T08:59:45.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.584+0000 7f246adc8700 1 --2- 172.21.15.103:0/3654356322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464133360 0x7f246409cc90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.584+0000 7f246adc8700 1 -- 172.21.15.103:0/3654356322 >> 172.21.15.103:0/3654356322 conn(0x7f24640a2080 msgr2=0x7f24640a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:45.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.585+0000 7f246adc8700 1 -- 172.21.15.103:0/3654356322 shutdown_connections 2023-07-02T08:59:45.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.585+0000 7f246adc8700 1 -- 172.21.15.103:0/3654356322 wait complete. 2023-07-02T08:59:45.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.585+0000 7f246adc8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:45.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.586+0000 7f246adc8700 1 Processor -- start 2023-07-02T08:59:45.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.586+0000 7f246adc8700 1 -- start start 2023-07-02T08:59:45.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.586+0000 7f246adc8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464132f50 0x7f24641ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.586+0000 7f246adc8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246406d370 con 0x7f2464132f50 2023-07-02T08:59:45.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f2468b64700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464132f50 0x7f24641ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f2468b64700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464132f50 0x7f24641ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36680/0 (socket says 172.21.15.103:36680) 2023-07-02T08:59:45.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f2468b64700 1 -- 172.21.15.103:0/1216838662 learned_addr learned my addr 172.21.15.103:0/1216838662 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:45.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f2468b64700 1 -- 172.21.15.103:0/1216838662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246406dde0 con 0x7f2464132f50 2023-07-02T08:59:45.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f2468b64700 1 --2- 172.21.15.103:0/1216838662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464132f50 0x7f24641ba9b0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f24617fa700 1 -- 172.21.15.103:0/1216838662 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2458004b80 con 0x7f2464132f50 2023-07-02T08:59:45.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f24617fa700 1 -- 172.21.15.103:0/1216838662 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2458004ce0 con 0x7f2464132f50 2023-07-02T08:59:45.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f24617fa700 1 -- 172.21.15.103:0/1216838662 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f245800c4f0 con 0x7f2464132f50 2023-07-02T08:59:45.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24641ba4f0 con 0x7f2464132f50 2023-07-02T08:59:45.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.587+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2464071e40 con 0x7f2464132f50 2023-07-02T08:59:45.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.578+0000 7f3b57446700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:45.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.588+0000 7f24617fa700 1 -- 172.21.15.103:0/1216838662 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f245800c650 con 0x7f2464132f50 2023-07-02T08:59:45.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.588+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2464061ea0 con 0x7f2464132f50 2023-07-02T08:59:45.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.588+0000 7f24617fa700 1 --2- 172.21.15.103:0/1216838662 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f24480377c0 0x7f2448039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.589+0000 7f24617fa700 1 -- 172.21.15.103:0/1216838662 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2458046970 con 0x7f2464132f50 2023-07-02T08:59:45.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.589+0000 7f2463fff700 1 --2- 172.21.15.103:0/1216838662 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f24480377c0 0x7f2448039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.589+0000 7f2463fff700 1 --2- 172.21.15.103:0/1216838662 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f24480377c0 0x7f2448039cd0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.590+0000 7f3b57446700 1 Processor -- start 2023-07-02T08:59:45.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.590+0000 7f3b57446700 1 -- start start 2023-07-02T08:59:45.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.591+0000 7f3b57446700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3b5012ee20 0x7f3b5012f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.591+0000 7f3b57446700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5009b0a0 con 0x7f3b50098a00 2023-07-02T08:59:45.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.591+0000 7f3b57446700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b500723f0 con 0x7f3b5012ee20 2023-07-02T08:59:45.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.591+0000 7f3b549e1700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3b50098a00 0x7f3b50098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55586/0 (socket says 172.21.15.103:55586) 2023-07-02T08:59:45.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.591+0000 7f3b549e1700 1 -- 172.21.15.103:0/598314863 learned_addr learned my addr 172.21.15.103:0/598314863 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:45.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.594+0000 7f3b551e2700 1 --2- 172.21.15.103:0/598314863 >> v2:172.21.15.103:3300/0 conn(0x7f3b5012ee20 0x7f3b5012f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.594+0000 7f3b47fff700 1 -- 172.21.15.103:0/598314863 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209464048 0 0) 0x7f3b5009b0a0 con 0x7f3b50098a00 2023-07-02T08:59:45.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.594+0000 7f3b47fff700 1 -- 172.21.15.103:0/598314863 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3c003480 con 0x7f3b50098a00 2023-07-02T08:59:45.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.594+0000 7f24617fa700 1 -- 172.21.15.103:0/1216838662 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2458009c80 con 0x7f2464132f50 2023-07-02T08:59:45.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.594+0000 7f3b551e2700 1 -- 172.21.15.103:0/598314863 >> v1:172.21.15.103:6789/0 conn(0x7f3b50098a00 legacy=0x7f3b50098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.594+0000 7f3b551e2700 1 -- 172.21.15.103:0/598314863 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5012e9d0 con 0x7f3b5012ee20 2023-07-02T08:59:45.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.597+0000 7f3b551e2700 1 --2- 172.21.15.103:0/598314863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b5012ee20 0x7f3b5012f230 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aba0b9a6db8cdd2 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.597+0000 7f3b47fff700 1 -- 172.21.15.103:0/598314863 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3b4000aae0 con 0x7f3b5012ee20 2023-07-02T08:59:45.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.597+0000 7f3b47fff700 1 -- 172.21.15.103:0/598314863 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3b4000ac40 con 0x7f3b5012ee20 2023-07-02T08:59:45.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.597+0000 7f3b47fff700 1 -- 172.21.15.103:0/598314863 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3b4000f3a0 con 0x7f3b5012ee20 2023-07-02T08:59:45.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.598+0000 7f3b57446700 1 -- 172.21.15.103:0/598314863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b5012ee20 msgr2=0x7f3b5012f230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.598+0000 7f3b57446700 1 --2- 172.21.15.103:0/598314863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b5012ee20 0x7f3b5012f230 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.598+0000 7f3b57446700 1 -- 172.21.15.103:0/598314863 shutdown_connections 2023-07-02T08:59:45.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.598+0000 7f3b57446700 1 --2- 172.21.15.103:0/598314863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b5012ee20 0x7f3b5012f230 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.598+0000 7f3b57446700 1 -- 172.21.15.103:0/598314863 >> 172.21.15.103:0/598314863 conn(0x7f3b5009d810 msgr2=0x7f3b5009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:45.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.598+0000 7f3b57446700 1 -- 172.21.15.103:0/598314863 shutdown_connections 2023-07-02T08:59:45.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.599+0000 7f3b57446700 1 -- 172.21.15.103:0/598314863 wait complete. 2023-07-02T08:59:45.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.599+0000 7f3b57446700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:45.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.600+0000 7f3b57446700 1 Processor -- start 2023-07-02T08:59:45.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.600+0000 7f3b57446700 1 -- start start 2023-07-02T08:59:45.603 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.602+0000 7f3b57446700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b50098a00 0x7f3b5011a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.603 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.602+0000 7f3b57446700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5011b9a0 con 0x7f3b50098a00 2023-07-02T08:59:45.603 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.602+0000 7f3b551e2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b50098a00 0x7f3b5011a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.603 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.602+0000 7f3b551e2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b50098a00 0x7f3b5011a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36696/0 (socket says 172.21.15.103:36696) 2023-07-02T08:59:45.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.602+0000 7f3b551e2700 1 -- 172.21.15.103:0/2738340030 learned_addr learned my addr 172.21.15.103:0/2738340030 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:45.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.602+0000 7f3b551e2700 1 -- 172.21.15.103:0/2738340030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5011ca80 con 0x7f3b50098a00 2023-07-02T08:59:45.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.603+0000 7f3b551e2700 1 --2- 172.21.15.103:0/2738340030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b50098a00 0x7f3b5011a4d0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.603+0000 7f3b45ffb700 1 -- 172.21.15.103:0/2738340030 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3b40015040 con 0x7f3b50098a00 2023-07-02T08:59:45.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.603+0000 7f3b45ffb700 1 -- 172.21.15.103:0/2738340030 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3b40010410 con 0x7f3b50098a00 2023-07-02T08:59:45.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.603+0000 7f3b45ffb700 1 -- 172.21.15.103:0/2738340030 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3b400106b0 con 0x7f3b50098a00 2023-07-02T08:59:45.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.603+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5011d6b0 con 0x7f3b50098a00 2023-07-02T08:59:45.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.603+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5011c130 con 0x7f3b50098a00 2023-07-02T08:59:45.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.604+0000 7f3b45ffb700 1 -- 172.21.15.103:0/2738340030 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3b40018040 con 0x7f3b50098a00 2023-07-02T08:59:45.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.604+0000 7f3b45ffb700 1 --2- 172.21.15.103:0/2738340030 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b340377b0 0x7f3b34039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.604+0000 7f3b45ffb700 1 -- 172.21.15.103:0/2738340030 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3b40048040 con 0x7f3b50098a00 2023-07-02T08:59:45.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.605+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5011e840 con 0x7f3b50098a00 2023-07-02T08:59:45.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.605+0000 7f3b549e1700 1 --2- 172.21.15.103:0/2738340030 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b340377b0 0x7f3b34039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.605+0000 7f3b549e1700 1 --2- 172.21.15.103:0/2738340030 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b340377b0 0x7f3b34039cc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.610+0000 7f3b45ffb700 1 -- 172.21.15.103:0/2738340030 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3b400182f0 con 0x7f3b50098a00 2023-07-02T08:59:45.646 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.640+0000 7f69496a0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:45.646 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.645+0000 7f69496a0700 1 Processor -- start 2023-07-02T08:59:45.647 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.645+0000 7f69496a0700 1 -- start start 2023-07-02T08:59:45.647 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.646+0000 7f69496a0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6944133360 0x7f694409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.647 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.646+0000 7f69496a0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694409e260 con 0x7f6944132f50 2023-07-02T08:59:45.648 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.646+0000 7f69496a0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69440a1640 con 0x7f6944133360 2023-07-02T08:59:45.648 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.646+0000 7f6942ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6944132f50 0x7f694409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55596/0 (socket says 172.21.15.103:55596) 2023-07-02T08:59:45.648 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.646+0000 7f69427fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6944133360 0x7f694409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.648 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.646+0000 7f6942ffd700 1 -- 172.21.15.103:0/3609407972 learned_addr learned my addr 172.21.15.103:0/3609407972 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:45.649 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.647+0000 7f69427fc700 1 -- 172.21.15.103:0/3609407972 >> v1:172.21.15.103:6789/0 conn(0x7f6944132f50 legacy=0x7f694409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.649 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.647+0000 7f69427fc700 1 -- 172.21.15.103:0/3609407972 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944132a20 con 0x7f6944133360 2023-07-02T08:59:45.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.647+0000 7f6941ffb700 1 -- 172.21.15.103:0/3609407972 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579837262 0 0) 0x7f694409e260 con 0x7f6944132f50 2023-07-02T08:59:45.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.647+0000 7f69427fc700 1 --2- 172.21.15.103:0/3609407972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944133360 0x7f694409cc90 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac73b5658f481dec server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.647+0000 7f6941ffb700 1 -- 172.21.15.103:0/3609407972 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6938009cd0 con 0x7f6944133360 2023-07-02T08:59:45.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.647+0000 7f6941ffb700 1 -- 172.21.15.103:0/3609407972 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6938009e30 con 0x7f6944133360 2023-07-02T08:59:45.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.647+0000 7f6941ffb700 1 -- 172.21.15.103:0/3609407972 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f693800e360 con 0x7f6944133360 2023-07-02T08:59:45.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.648+0000 7f69496a0700 1 -- 172.21.15.103:0/3609407972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944133360 msgr2=0x7f694409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.648+0000 7f69496a0700 1 --2- 172.21.15.103:0/3609407972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944133360 0x7f694409cc90 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.648+0000 7f69496a0700 1 -- 172.21.15.103:0/3609407972 shutdown_connections 2023-07-02T08:59:45.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.648+0000 7f69496a0700 1 --2- 172.21.15.103:0/3609407972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944133360 0x7f694409cc90 crc :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.648+0000 7f69496a0700 1 -- 172.21.15.103:0/3609407972 >> 172.21.15.103:0/3609407972 conn(0x7f69440a2080 msgr2=0x7f69440a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:45.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.648+0000 7f69496a0700 1 -- 172.21.15.103:0/3609407972 shutdown_connections 2023-07-02T08:59:45.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.648+0000 7f69496a0700 1 -- 172.21.15.103:0/3609407972 wait complete. 2023-07-02T08:59:45.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.648+0000 7f69496a0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:45.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.650+0000 7f69496a0700 1 Processor -- start 2023-07-02T08:59:45.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.650+0000 7f69496a0700 1 -- start start 2023-07-02T08:59:45.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.651+0000 7f69496a0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944132f50 0x7f694412b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.651+0000 7f69496a0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944133a00 con 0x7f6944132f50 2023-07-02T08:59:45.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.651+0000 7f6942ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944132f50 0x7f694412b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.651+0000 7f6942ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944132f50 0x7f694412b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36716/0 (socket says 172.21.15.103:36716) 2023-07-02T08:59:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.651+0000 7f6942ffd700 1 -- 172.21.15.103:0/1227370429 learned_addr learned my addr 172.21.15.103:0/1227370429 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.652+0000 7f6942ffd700 1 -- 172.21.15.103:0/1227370429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694406f9e0 con 0x7f6944132f50 2023-07-02T08:59:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.652+0000 7f6942ffd700 1 --2- 172.21.15.103:0/1227370429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944132f50 0x7f694412b540 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.652+0000 7f692b7fe700 1 -- 172.21.15.103:0/1227370429 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6934005a60 con 0x7f6944132f50 2023-07-02T08:59:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.652+0000 7f692b7fe700 1 -- 172.21.15.103:0/1227370429 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6934005bc0 con 0x7f6944132f50 2023-07-02T08:59:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.652+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694406da90 con 0x7f6944132f50 2023-07-02T08:59:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.652+0000 7f692b7fe700 1 -- 172.21.15.103:0/1227370429 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6934005e60 con 0x7f6944132f50 2023-07-02T08:59:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.652+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694406f1d0 con 0x7f6944132f50 2023-07-02T08:59:45.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.653+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6944061ea0 con 0x7f6944132f50 2023-07-02T08:59:45.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.653+0000 7f692b7fe700 1 -- 172.21.15.103:0/1227370429 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f69340070b0 con 0x7f6944132f50 2023-07-02T08:59:45.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.654+0000 7f692b7fe700 1 --2- 172.21.15.103:0/1227370429 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6930037770 0x7f6930039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:45.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.654+0000 7f692b7fe700 1 -- 172.21.15.103:0/1227370429 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6934046ae0 con 0x7f6944132f50 2023-07-02T08:59:45.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.654+0000 7f69427fc700 1 --2- 172.21.15.103:0/1227370429 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6930037770 0x7f6930039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:45.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.655+0000 7f69427fc700 1 --2- 172.21.15.103:0/1227370429 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6930037770 0x7f6930039c80 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:45.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.658+0000 7f692b7fe700 1 -- 172.21.15.103:0/1227370429 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f693400a300 con 0x7f6944132f50 2023-07-02T08:59:45.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.784+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc84006e070 con 0x7fc84009c580 2023-07-02T08:59:45.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.785+0000 7fc83cff9700 1 -- 172.21.15.103:0/288803802 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fc8300189f0 con 0x7fc84009c580 2023-07-02T08:59:45.786 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:45.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc82c0377c0 msgr2=0x7fc82c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 --2- 172.21.15.103:0/288803802 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc82c0377c0 0x7fc82c039cd0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc84009c580 msgr2=0x7fc84012c0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 --2- 172.21.15.103:0/288803802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc84009c580 0x7fc84012c0f0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 shutdown_connections 2023-07-02T08:59:45.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 --2- 172.21.15.103:0/288803802 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc82c0377c0 0x7fc82c039cd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 --2- 172.21.15.103:0/288803802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc84009c580 0x7fc84012c0f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 >> 172.21.15.103:0/288803802 conn(0x7fc8400a2080 msgr2=0x7fc8401bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:45.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 shutdown_connections 2023-07-02T08:59:45.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.787+0000 7fc8464fd700 1 -- 172.21.15.103:0/288803802 wait complete. 2023-07-02T08:59:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.795+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f24640725b0 con 0x7f2464132f50 2023-07-02T08:59:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.795+0000 7f24617fa700 1 -- 172.21.15.103:0/1216838662 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f24580098a0 con 0x7f2464132f50 2023-07-02T08:59:45.797 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f24480377c0 msgr2=0x7f2448039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 --2- 172.21.15.103:0/1216838662 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f24480377c0 0x7f2448039cd0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464132f50 msgr2=0x7f24641ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 --2- 172.21.15.103:0/1216838662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464132f50 0x7f24641ba9b0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 shutdown_connections 2023-07-02T08:59:45.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 --2- 172.21.15.103:0/1216838662 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f24480377c0 0x7f2448039cd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 --2- 172.21.15.103:0/1216838662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2464132f50 0x7f24641ba9b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 >> 172.21.15.103:0/1216838662 conn(0x7f24640a2080 msgr2=0x7f24640a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:45.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 shutdown_connections 2023-07-02T08:59:45.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.798+0000 7f246adc8700 1 -- 172.21.15.103:0/1216838662 wait complete. 2023-07-02T08:59:45.805 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:45.813 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:45.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.817+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3b50061ea0 con 0x7f3b50098a00 2023-07-02T08:59:45.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.817+0000 7f3b45ffb700 1 -- 172.21.15.103:0/2738340030 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3b4000da60 con 0x7f3b50098a00 2023-07-02T08:59:45.819 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:45.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b340377b0 msgr2=0x7f3b34039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 --2- 172.21.15.103:0/2738340030 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b340377b0 0x7f3b34039cc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b50098a00 msgr2=0x7f3b5011a4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 --2- 172.21.15.103:0/2738340030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b50098a00 0x7f3b5011a4d0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 shutdown_connections 2023-07-02T08:59:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 --2- 172.21.15.103:0/2738340030 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3b340377b0 0x7f3b34039cc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 --2- 172.21.15.103:0/2738340030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b50098a00 0x7f3b5011a4d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 >> 172.21.15.103:0/2738340030 conn(0x7f3b5009d810 msgr2=0x7f3b5012cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 shutdown_connections 2023-07-02T08:59:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.819+0000 7f3b57446700 1 -- 172.21.15.103:0/2738340030 wait complete. 2023-07-02T08:59:45.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.859+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f69440725b0 con 0x7f6944132f50 2023-07-02T08:59:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.860+0000 7f692b7fe700 1 -- 172.21.15.103:0/1227370429 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f693400a120 con 0x7f6944132f50 2023-07-02T08:59:45.861 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:45.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6930037770 msgr2=0x7f6930039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 --2- 172.21.15.103:0/1227370429 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6930037770 0x7f6930039c80 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944132f50 msgr2=0x7f694412b540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:45.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 --2- 172.21.15.103:0/1227370429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944132f50 0x7f694412b540 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 shutdown_connections 2023-07-02T08:59:45.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 --2- 172.21.15.103:0/1227370429 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6930037770 0x7f6930039c80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 --2- 172.21.15.103:0/1227370429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6944132f50 0x7f694412b540 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:45.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 >> 172.21.15.103:0/1227370429 conn(0x7f69440a2080 msgr2=0x7f69441bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:45.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 shutdown_connections 2023-07-02T08:59:45.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:45.862+0000 7f69496a0700 1 -- 172.21.15.103:0/1227370429 wait complete. 2023-07-02T08:59:45.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:45.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-07-02T08:59:46.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:46.814 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:46.835 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:46.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:46.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.957+0000 7fd765ec8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:46.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.963+0000 7fd765ec8700 1 Processor -- start 2023-07-02T08:59:46.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.963+0000 7fd765ec8700 1 -- start start 2023-07-02T08:59:46.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.963+0000 7fd765ec8700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd760133360 0x7fd76009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:46.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.963+0000 7fd765ec8700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76009e260 con 0x7fd760132f50 2023-07-02T08:59:46.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.963+0000 7fd765ec8700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600a1640 con 0x7fd760133360 2023-07-02T08:59:46.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.963+0000 7fd75effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd760133360 0x7fd76009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:46.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd760133360 0x7fd76009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36726/0 (socket says 172.21.15.103:36726) 2023-07-02T08:59:46.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75effd700 1 -- 172.21.15.103:0/1992314000 learned_addr learned my addr 172.21.15.103:0/1992314000 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:46.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75f7fe700 1 --1- 172.21.15.103:0/1992314000 >> v1:172.21.15.103:6789/0 conn(0x7fd760132f50 0x7fd76009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55610/0 (socket says 172.21.15.103:55610) 2023-07-02T08:59:46.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75effd700 1 -- 172.21.15.103:0/1992314000 >> v1:172.21.15.103:6789/0 conn(0x7fd760132f50 legacy=0x7fd76009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:46.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75effd700 1 -- 172.21.15.103:0/1992314000 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd760132a20 con 0x7fd760133360 2023-07-02T08:59:46.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75effd700 1 --2- 172.21.15.103:0/1992314000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760133360 0x7fd76009cc90 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5ff5c80e867a78 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:46.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75e7fc700 1 -- 172.21.15.103:0/1992314000 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd754009d90 con 0x7fd760133360 2023-07-02T08:59:46.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75e7fc700 1 -- 172.21.15.103:0/1992314000 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd75400e230 con 0x7fd760133360 2023-07-02T08:59:46.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.964+0000 7fd75e7fc700 1 -- 172.21.15.103:0/1992314000 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd75400e4d0 con 0x7fd760133360 2023-07-02T08:59:46.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.965+0000 7fd765ec8700 1 -- 172.21.15.103:0/1992314000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760133360 msgr2=0x7fd76009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:46.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.965+0000 7fd765ec8700 1 --2- 172.21.15.103:0/1992314000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760133360 0x7fd76009cc90 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:46.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.965+0000 7fd765ec8700 1 -- 172.21.15.103:0/1992314000 shutdown_connections 2023-07-02T08:59:46.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.965+0000 7fd765ec8700 1 --2- 172.21.15.103:0/1992314000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760133360 0x7fd76009cc90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:46.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.965+0000 7fd765ec8700 1 -- 172.21.15.103:0/1992314000 >> 172.21.15.103:0/1992314000 conn(0x7fd7600a2080 msgr2=0x7fd7600a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:46.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.965+0000 7fd765ec8700 1 -- 172.21.15.103:0/1992314000 shutdown_connections 2023-07-02T08:59:46.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.965+0000 7fd765ec8700 1 -- 172.21.15.103:0/1992314000 wait complete. 2023-07-02T08:59:46.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.965+0000 7fd765ec8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:46.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.966+0000 7fd765ec8700 1 Processor -- start 2023-07-02T08:59:46.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.967+0000 7fd765ec8700 1 -- start start 2023-07-02T08:59:46.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.967+0000 7fd765ec8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760132f50 0x7fd76012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:46.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.967+0000 7fd765ec8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76006d260 con 0x7fd760132f50 2023-07-02T08:59:46.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.967+0000 7fd75f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760132f50 0x7fd76012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:46.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.967+0000 7fd75f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760132f50 0x7fd76012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36738/0 (socket says 172.21.15.103:36738) 2023-07-02T08:59:46.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.967+0000 7fd75f7fe700 1 -- 172.21.15.103:0/130580416 learned_addr learned my addr 172.21.15.103:0/130580416 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.968+0000 7fd75f7fe700 1 -- 172.21.15.103:0/130580416 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76012bb90 con 0x7fd760132f50 2023-07-02T08:59:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.968+0000 7fd75f7fe700 1 --2- 172.21.15.103:0/130580416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760132f50 0x7fd76012bea0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.968+0000 7fd747fff700 1 -- 172.21.15.103:0/130580416 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd7500059a0 con 0x7fd760132f50 2023-07-02T08:59:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.968+0000 7fd747fff700 1 -- 172.21.15.103:0/130580416 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd750005b00 con 0x7fd760132f50 2023-07-02T08:59:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.968+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76006f730 con 0x7fd760132f50 2023-07-02T08:59:46.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.968+0000 7fd747fff700 1 -- 172.21.15.103:0/130580416 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd750005da0 con 0x7fd760132f50 2023-07-02T08:59:46.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.968+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76006f490 con 0x7fd760132f50 2023-07-02T08:59:46.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.969+0000 7fd747fff700 1 -- 172.21.15.103:0/130580416 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd7500070b0 con 0x7fd760132f50 2023-07-02T08:59:46.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.969+0000 7fd747fff700 1 --2- 172.21.15.103:0/130580416 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd74c037770 0x7fd74c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:46.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.969+0000 7fd747fff700 1 -- 172.21.15.103:0/130580416 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd750046a40 con 0x7fd760132f50 2023-07-02T08:59:46.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.969+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76006f490 con 0x7fd760132f50 2023-07-02T08:59:46.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.970+0000 7fd75effd700 1 --2- 172.21.15.103:0/130580416 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd74c037770 0x7fd74c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:46.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.970+0000 7fd75effd700 1 --2- 172.21.15.103:0/130580416 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd74c037770 0x7fd74c039c80 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.979+0000 7fd747fff700 1 -- 172.21.15.103:0/130580416 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd76006f490 con 0x7fd760132f50 2023-07-02T08:59:46.999 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.993+0000 7fdf49621700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:46.999 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.998+0000 7fdf49621700 1 Processor -- start 2023-07-02T08:59:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.999+0000 7fdf49621700 1 -- start start 2023-07-02T08:59:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.999+0000 7fdf49621700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf44133360 0x7fdf4409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.999+0000 7fdf49621700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4409e260 con 0x7fdf44132f50 2023-07-02T08:59:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.999+0000 7fdf49621700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf440a1640 con 0x7fdf44133360 2023-07-02T08:59:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.999+0000 7fdf427fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf44133360 0x7fdf4409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf427fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf44133360 0x7fdf4409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36752/0 (socket says 172.21.15.103:36752) 2023-07-02T08:59:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf427fc700 1 -- 172.21.15.103:0/707550740 learned_addr learned my addr 172.21.15.103:0/707550740 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf42ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdf44132f50 0x7fdf4409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55612/0 (socket says 172.21.15.103:55612) 2023-07-02T08:59:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf427fc700 1 -- 172.21.15.103:0/707550740 >> v1:172.21.15.103:6789/0 conn(0x7fdf44132f50 legacy=0x7fdf4409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf427fc700 1 -- 172.21.15.103:0/707550740 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44132a20 con 0x7fdf44133360 2023-07-02T08:59:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf427fc700 1 --2- 172.21.15.103:0/707550740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44133360 0x7fdf4409cc90 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef34ce822af3ff14 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf41ffb700 1 -- 172.21.15.103:0/707550740 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf38009d90 con 0x7fdf44133360 2023-07-02T08:59:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf41ffb700 1 -- 172.21.15.103:0/707550740 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdf3800e230 con 0x7fdf44133360 2023-07-02T08:59:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.000+0000 7fdf41ffb700 1 -- 172.21.15.103:0/707550740 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf3800e4d0 con 0x7fdf44133360 2023-07-02T08:59:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.001+0000 7fdf49621700 1 -- 172.21.15.103:0/707550740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44133360 msgr2=0x7fdf4409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.001+0000 7fdf49621700 1 --2- 172.21.15.103:0/707550740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44133360 0x7fdf4409cc90 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.001+0000 7fdf49621700 1 -- 172.21.15.103:0/707550740 shutdown_connections 2023-07-02T08:59:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.001+0000 7fdf49621700 1 --2- 172.21.15.103:0/707550740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44133360 0x7fdf4409cc90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.001+0000 7fdf49621700 1 -- 172.21.15.103:0/707550740 >> 172.21.15.103:0/707550740 conn(0x7fdf440a2080 msgr2=0x7fdf440a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.001+0000 7fdf49621700 1 -- 172.21.15.103:0/707550740 shutdown_connections 2023-07-02T08:59:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.001+0000 7fdf49621700 1 -- 172.21.15.103:0/707550740 wait complete. 2023-07-02T08:59:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.002+0000 7fdf49621700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:47.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.003+0000 7fdf49621700 1 Processor -- start 2023-07-02T08:59:47.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.003+0000 7fdf49621700 1 -- start start 2023-07-02T08:59:47.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.004+0000 7fdf49621700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44132f50 0x7fdf4412bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.004+0000 7fdf49621700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4406d260 con 0x7fdf44132f50 2023-07-02T08:59:47.006 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.004+0000 7fdf42ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44132f50 0x7fdf4412bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.004+0000 7fdf42ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44132f50 0x7fdf4412bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36758/0 (socket says 172.21.15.103:36758) 2023-07-02T08:59:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.004+0000 7fdf42ffd700 1 -- 172.21.15.103:0/2475537397 learned_addr learned my addr 172.21.15.103:0/2475537397 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.005+0000 7fdf42ffd700 1 -- 172.21.15.103:0/2475537397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4412bb90 con 0x7fdf44132f50 2023-07-02T08:59:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.005+0000 7fdf42ffd700 1 --2- 172.21.15.103:0/2475537397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44132f50 0x7fdf4412bea0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.005+0000 7fdf2b7fe700 1 -- 172.21.15.103:0/2475537397 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf340059a0 con 0x7fdf44132f50 2023-07-02T08:59:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.005+0000 7fdf2b7fe700 1 -- 172.21.15.103:0/2475537397 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdf34005b00 con 0x7fdf44132f50 2023-07-02T08:59:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.005+0000 7fdf2b7fe700 1 -- 172.21.15.103:0/2475537397 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf34005da0 con 0x7fdf44132f50 2023-07-02T08:59:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.005+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4406f730 con 0x7fdf44132f50 2023-07-02T08:59:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:46.997+0000 7ffba2849700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.005+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4406f490 con 0x7fdf44132f50 2023-07-02T08:59:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.006+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf44061ea0 con 0x7fdf44132f50 2023-07-02T08:59:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.007+0000 7fdf2b7fe700 1 -- 172.21.15.103:0/2475537397 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fdf340070b0 con 0x7fdf44132f50 2023-07-02T08:59:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.007+0000 7fdf2b7fe700 1 --2- 172.21.15.103:0/2475537397 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf30037770 0x7fdf30039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7fdf2b7fe700 1 -- 172.21.15.103:0/2475537397 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fdf34046c00 con 0x7fdf44132f50 2023-07-02T08:59:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.007+0000 7ffba2849700 1 Processor -- start 2023-07-02T08:59:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.007+0000 7ffba2849700 1 -- start start 2023-07-02T08:59:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffba2849700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffb9c0717c0 0x7ffb9c132f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffba2849700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c132990 con 0x7ffb9c133450 2023-07-02T08:59:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffba2849700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c0715d0 con 0x7ffb9c0717c0 2023-07-02T08:59:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7fdf427fc700 1 --2- 172.21.15.103:0/2475537397 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf30037770 0x7fdf30039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffb9bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffb9c0717c0 0x7ffb9c132f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffb9b7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ffb9c133450 0x7ffb9c06f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55622/0 (socket says 172.21.15.103:55622) 2023-07-02T08:59:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffb9bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffb9c0717c0 0x7ffb9c132f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36764/0 (socket says 172.21.15.103:36764) 2023-07-02T08:59:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffb9b7fe700 1 -- 172.21.15.103:0/3464312597 learned_addr learned my addr 172.21.15.103:0/3464312597 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffb9bfff700 1 -- 172.21.15.103:0/3464312597 >> v1:172.21.15.103:6789/0 conn(0x7ffb9c133450 legacy=0x7ffb9c06f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffb9bfff700 1 -- 172.21.15.103:0/3464312597 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c070a50 con 0x7ffb9c0717c0 2023-07-02T08:59:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7ffb9bfff700 1 --2- 172.21.15.103:0/3464312597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c132f10 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ece88a1179e8470 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.008+0000 7fdf427fc700 1 --2- 172.21.15.103:0/2475537397 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf30037770 0x7fdf30039c80 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.016+0000 7ffb9affd700 1 -- 172.21.15.103:0/3464312597 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ffb9400e040 con 0x7ffb9c0717c0 2023-07-02T08:59:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.016+0000 7ffb9affd700 1 -- 172.21.15.103:0/3464312597 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ffb94007670 con 0x7ffb9c0717c0 2023-07-02T08:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.016+0000 7ffb9affd700 1 -- 172.21.15.103:0/3464312597 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ffb94007910 con 0x7ffb9c0717c0 2023-07-02T08:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.016+0000 7fdf2b7fe700 1 -- 172.21.15.103:0/2475537397 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fdf3400d710 con 0x7fdf44132f50 2023-07-02T08:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.017+0000 7ffba2849700 1 -- 172.21.15.103:0/3464312597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 msgr2=0x7ffb9c132f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.017+0000 7ffba2849700 1 --2- 172.21.15.103:0/3464312597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c132f10 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.017+0000 7ffba2849700 1 -- 172.21.15.103:0/3464312597 shutdown_connections 2023-07-02T08:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.017+0000 7ffba2849700 1 --2- 172.21.15.103:0/3464312597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c132f10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.017+0000 7ffba2849700 1 -- 172.21.15.103:0/3464312597 >> 172.21.15.103:0/3464312597 conn(0x7ffb9c0a22b0 msgr2=0x7ffb9c06d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.017+0000 7ffba2849700 1 -- 172.21.15.103:0/3464312597 shutdown_connections 2023-07-02T08:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.018+0000 7ffba2849700 1 -- 172.21.15.103:0/3464312597 wait complete. 2023-07-02T08:59:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.018+0000 7ffba2849700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.019+0000 7ffba2849700 1 Processor -- start 2023-07-02T08:59:47.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.019+0000 7ffba2849700 1 -- start start 2023-07-02T08:59:47.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffba2849700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c095040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffba2849700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c096d00 con 0x7ffb9c0717c0 2023-07-02T08:59:47.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffb9bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c095040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffb9bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c095040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36776/0 (socket says 172.21.15.103:36776) 2023-07-02T08:59:47.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffb9bfff700 1 -- 172.21.15.103:0/4237486542 learned_addr learned my addr 172.21.15.103:0/4237486542 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:47.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffb9bfff700 1 -- 172.21.15.103:0/4237486542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c097960 con 0x7ffb9c0717c0 2023-07-02T08:59:47.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffb9bfff700 1 --2- 172.21.15.103:0/4237486542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c095040 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffb98ff9700 1 -- 172.21.15.103:0/4237486542 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ffb9400e040 con 0x7ffb9c0717c0 2023-07-02T08:59:47.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.021+0000 7ffb98ff9700 1 -- 172.21.15.103:0/4237486542 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ffb94013a90 con 0x7ffb9c0717c0 2023-07-02T08:59:47.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.022+0000 7ffb98ff9700 1 -- 172.21.15.103:0/4237486542 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ffb94013d30 con 0x7ffb9c0717c0 2023-07-02T08:59:47.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.022+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c0987d0 con 0x7ffb9c0717c0 2023-07-02T08:59:47.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.022+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c098370 con 0x7ffb9c0717c0 2023-07-02T08:59:47.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.023+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9c061ea0 con 0x7ffb9c0717c0 2023-07-02T08:59:47.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.024+0000 7ffb98ff9700 1 -- 172.21.15.103:0/4237486542 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ffb94011040 con 0x7ffb9c0717c0 2023-07-02T08:59:47.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.024+0000 7ffb98ff9700 1 --2- 172.21.15.103:0/4237486542 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ffb7c0377c0 0x7ffb7c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.024+0000 7ffb98ff9700 1 -- 172.21.15.103:0/4237486542 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ffb94048540 con 0x7ffb9c0717c0 2023-07-02T08:59:47.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.024+0000 7ffb9b7fe700 1 --2- 172.21.15.103:0/4237486542 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ffb7c0377c0 0x7ffb7c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.025+0000 7ffb9b7fe700 1 --2- 172.21.15.103:0/4237486542 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ffb7c0377c0 0x7ffb7c039cd0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.033+0000 7ffb98ff9700 1 -- 172.21.15.103:0/4237486542 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ffb94016e40 con 0x7ffb9c0717c0 2023-07-02T08:59:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.042+0000 7fa14e88d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:47.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.048+0000 7fa14e88d700 1 Processor -- start 2023-07-02T08:59:47.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.048+0000 7fa14e88d700 1 -- start start 2023-07-02T08:59:47.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.048+0000 7fa14e88d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa148132f50 0x7fa148133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.048+0000 7fa14e88d700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14809e260 con 0x7fa14809c580 2023-07-02T08:59:47.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.048+0000 7fa14e88d700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1480a1640 con 0x7fa148132f50 2023-07-02T08:59:47.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa147fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa148132f50 0x7fa148133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa1477fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa14809c580 0x7fa14809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55638/0 (socket says 172.21.15.103:55638) 2023-07-02T08:59:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa1477fe700 1 -- 172.21.15.103:0/2095411734 learned_addr learned my addr 172.21.15.103:0/2095411734 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa147fff700 1 -- 172.21.15.103:0/2095411734 >> v1:172.21.15.103:6789/0 conn(0x7fa14809c580 legacy=0x7fa14809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa147fff700 1 -- 172.21.15.103:0/2095411734 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa148132a20 con 0x7fa148132f50 2023-07-02T08:59:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa147fff700 1 --2- 172.21.15.103:0/2095411734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa148132f50 0x7fa148133360 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f93e903910c3fc server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa146ffd700 1 -- 172.21.15.103:0/2095411734 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa138009cd0 con 0x7fa148132f50 2023-07-02T08:59:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa146ffd700 1 -- 172.21.15.103:0/2095411734 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa138009e30 con 0x7fa148132f50 2023-07-02T08:59:47.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.049+0000 7fa146ffd700 1 -- 172.21.15.103:0/2095411734 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa13800e360 con 0x7fa148132f50 2023-07-02T08:59:47.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.050+0000 7fa14e88d700 1 -- 172.21.15.103:0/2095411734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa148132f50 msgr2=0x7fa148133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.050+0000 7fa14e88d700 1 --2- 172.21.15.103:0/2095411734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa148132f50 0x7fa148133360 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.050+0000 7fa14e88d700 1 -- 172.21.15.103:0/2095411734 shutdown_connections 2023-07-02T08:59:47.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.050+0000 7fa14e88d700 1 --2- 172.21.15.103:0/2095411734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa148132f50 0x7fa148133360 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.050+0000 7fa14e88d700 1 -- 172.21.15.103:0/2095411734 >> 172.21.15.103:0/2095411734 conn(0x7fa1480a2080 msgr2=0x7fa1480a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:47.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.050+0000 7fa14e88d700 1 -- 172.21.15.103:0/2095411734 shutdown_connections 2023-07-02T08:59:47.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.050+0000 7fa14e88d700 1 -- 172.21.15.103:0/2095411734 wait complete. 2023-07-02T08:59:47.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.050+0000 7fa14e88d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:47.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.051+0000 7fa14e88d700 1 Processor -- start 2023-07-02T08:59:47.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.051+0000 7fa14e88d700 1 -- start start 2023-07-02T08:59:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.052+0000 7fa14e88d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa14809c580 0x7fa14812c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.052+0000 7fa14e88d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14812bc00 con 0x7fa14809c580 2023-07-02T08:59:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.052+0000 7fa147fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa14809c580 0x7fa14812c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.052+0000 7fa147fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa14809c580 0x7fa14812c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36792/0 (socket says 172.21.15.103:36792) 2023-07-02T08:59:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.052+0000 7fa147fff700 1 -- 172.21.15.103:0/2386039038 learned_addr learned my addr 172.21.15.103:0/2386039038 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.053+0000 7fa147fff700 1 -- 172.21.15.103:0/2386039038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa148133a70 con 0x7fa14809c580 2023-07-02T08:59:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.053+0000 7fa147fff700 1 --2- 172.21.15.103:0/2386039038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa14809c580 0x7fa14812c050 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.053+0000 7fa144ff9700 1 -- 172.21.15.103:0/2386039038 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa138013530 con 0x7fa14809c580 2023-07-02T08:59:47.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.053+0000 7fa144ff9700 1 -- 172.21.15.103:0/2386039038 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa138013690 con 0x7fa14809c580 2023-07-02T08:59:47.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.053+0000 7fa144ff9700 1 -- 172.21.15.103:0/2386039038 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa138013930 con 0x7fa14809c580 2023-07-02T08:59:47.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.054+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14806dfe0 con 0x7fa14809c580 2023-07-02T08:59:47.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.054+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14806f7d0 con 0x7fa14809c580 2023-07-02T08:59:47.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.055+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14812ba20 con 0x7fa14809c580 2023-07-02T08:59:47.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.055+0000 7fa144ff9700 1 -- 172.21.15.103:0/2386039038 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa138013ae0 con 0x7fa14809c580 2023-07-02T08:59:47.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.055+0000 7fa144ff9700 1 --2- 172.21.15.103:0/2386039038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1340377c0 0x7fa134039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:47.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.055+0000 7fa144ff9700 1 -- 172.21.15.103:0/2386039038 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa138018040 con 0x7fa14809c580 2023-07-02T08:59:47.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.057+0000 7fa1477fe700 1 --2- 172.21.15.103:0/2386039038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1340377c0 0x7fa134039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:47.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.057+0000 7fa1477fe700 1 --2- 172.21.15.103:0/2386039038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1340377c0 0x7fa134039cd0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:47.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.060+0000 7fa144ff9700 1 -- 172.21.15.103:0/2386039038 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa138018a80 con 0x7fa14809c580 2023-07-02T08:59:47.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.198+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd76006f490 con 0x7fd760132f50 2023-07-02T08:59:47.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.199+0000 7fd747fff700 1 -- 172.21.15.103:0/130580416 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd76006f490 con 0x7fd760132f50 2023-07-02T08:59:47.200 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:47.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd74c037770 msgr2=0x7fd74c039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 --2- 172.21.15.103:0/130580416 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd74c037770 0x7fd74c039c80 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760132f50 msgr2=0x7fd76012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 --2- 172.21.15.103:0/130580416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760132f50 0x7fd76012bea0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 shutdown_connections 2023-07-02T08:59:47.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 --2- 172.21.15.103:0/130580416 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd74c037770 0x7fd74c039c80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 --2- 172.21.15.103:0/130580416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd760132f50 0x7fd76012bea0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 >> 172.21.15.103:0/130580416 conn(0x7fd7600a2080 msgr2=0x7fd7601bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:47.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 shutdown_connections 2023-07-02T08:59:47.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.201+0000 7fd765ec8700 1 -- 172.21.15.103:0/130580416 wait complete. 2023-07-02T08:59:47.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:47.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.221+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fdf4406f490 con 0x7fdf44132f50 2023-07-02T08:59:47.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.221+0000 7fdf2b7fe700 1 -- 172.21.15.103:0/2475537397 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fdf4406f490 con 0x7fdf44132f50 2023-07-02T08:59:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:47.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf30037770 msgr2=0x7fdf30039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 --2- 172.21.15.103:0/2475537397 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf30037770 0x7fdf30039c80 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44132f50 msgr2=0x7fdf4412bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 --2- 172.21.15.103:0/2475537397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44132f50 0x7fdf4412bea0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 shutdown_connections 2023-07-02T08:59:47.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 --2- 172.21.15.103:0/2475537397 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf30037770 0x7fdf30039c80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 --2- 172.21.15.103:0/2475537397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf44132f50 0x7fdf4412bea0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 >> 172.21.15.103:0/2475537397 conn(0x7fdf440a2080 msgr2=0x7fdf441bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:47.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 shutdown_connections 2023-07-02T08:59:47.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.223+0000 7fdf49621700 1 -- 172.21.15.103:0/2475537397 wait complete. 2023-07-02T08:59:47.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.234+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ffb9c073ca0 con 0x7ffb9c0717c0 2023-07-02T08:59:47.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.234+0000 7ffb98ff9700 1 -- 172.21.15.103:0/4237486542 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ffb94010a70 con 0x7ffb9c0717c0 2023-07-02T08:59:47.236 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:47.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ffb7c0377c0 msgr2=0x7ffb7c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 --2- 172.21.15.103:0/4237486542 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ffb7c0377c0 0x7ffb7c039cd0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 msgr2=0x7ffb9c095040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 --2- 172.21.15.103:0/4237486542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c095040 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 shutdown_connections 2023-07-02T08:59:47.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 --2- 172.21.15.103:0/4237486542 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ffb7c0377c0 0x7ffb7c039cd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 --2- 172.21.15.103:0/4237486542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb9c0717c0 0x7ffb9c095040 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 >> 172.21.15.103:0/4237486542 conn(0x7ffb9c0a22b0 msgr2=0x7ffb9c131a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:47.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 shutdown_connections 2023-07-02T08:59:47.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.236+0000 7ffba2849700 1 -- 172.21.15.103:0/4237486542 wait complete. 2023-07-02T08:59:47.240 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:47.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:47.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.262+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa148133a70 con 0x7fa14809c580 2023-07-02T08:59:47.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.262+0000 7fa144ff9700 1 -- 172.21.15.103:0/2386039038 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa148133a70 con 0x7fa14809c580 2023-07-02T08:59:47.264 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:47.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.264+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1340377c0 msgr2=0x7fa134039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.264+0000 7fa14e88d700 1 --2- 172.21.15.103:0/2386039038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1340377c0 0x7fa134039cd0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.264+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa14809c580 msgr2=0x7fa14812c050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:47.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.264+0000 7fa14e88d700 1 --2- 172.21.15.103:0/2386039038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa14809c580 0x7fa14812c050 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.264+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 shutdown_connections 2023-07-02T08:59:47.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.264+0000 7fa14e88d700 1 --2- 172.21.15.103:0/2386039038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1340377c0 0x7fa134039cd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.265+0000 7fa14e88d700 1 --2- 172.21.15.103:0/2386039038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa14809c580 0x7fa14812c050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:47.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.265+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 >> 172.21.15.103:0/2386039038 conn(0x7fa1480a2080 msgr2=0x7fa1481bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:47.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.265+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 shutdown_connections 2023-07-02T08:59:47.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:47.265+0000 7fa14e88d700 1 -- 172.21.15.103:0/2386039038 wait complete. 2023-07-02T08:59:47.279 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:48.218 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:48.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:48.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:48.280 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:48.367 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.360+0000 7f2359b8c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:48.368 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.367+0000 7f2359b8c700 1 Processor -- start 2023-07-02T08:59:48.368 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.367+0000 7f2359b8c700 1 -- start start 2023-07-02T08:59:48.369 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2359b8c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2354133360 0x7f235409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.369 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2359b8c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235409e260 con 0x7f2354132f50 2023-07-02T08:59:48.369 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2359b8c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23540a1640 con 0x7f2354133360 2023-07-02T08:59:48.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2352ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2354133360 0x7f235409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2352ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2354133360 0x7f235409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36806/0 (socket says 172.21.15.103:36806) 2023-07-02T08:59:48.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2352ffd700 1 -- 172.21.15.103:0/3922952242 learned_addr learned my addr 172.21.15.103:0/3922952242 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:48.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f23537fe700 1 --1- 172.21.15.103:0/3922952242 >> v1:172.21.15.103:6789/0 conn(0x7f2354132f50 0x7f235409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55644/0 (socket says 172.21.15.103:55644) 2023-07-02T08:59:48.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2352ffd700 1 -- 172.21.15.103:0/3922952242 >> v1:172.21.15.103:6789/0 conn(0x7f2354132f50 legacy=0x7f235409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2352ffd700 1 -- 172.21.15.103:0/3922952242 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354132a20 con 0x7f2354133360 2023-07-02T08:59:48.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.368+0000 7f2352ffd700 1 --2- 172.21.15.103:0/3922952242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354133360 0x7f235409cc90 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a03db0dfe3d069 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.369+0000 7f23527fc700 1 -- 172.21.15.103:0/3922952242 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2348009d90 con 0x7f2354133360 2023-07-02T08:59:48.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.369+0000 7f23527fc700 1 -- 172.21.15.103:0/3922952242 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f234800e230 con 0x7f2354133360 2023-07-02T08:59:48.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.369+0000 7f23527fc700 1 -- 172.21.15.103:0/3922952242 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f234800e4d0 con 0x7f2354133360 2023-07-02T08:59:48.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.369+0000 7f2359b8c700 1 -- 172.21.15.103:0/3922952242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354133360 msgr2=0x7f235409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.369+0000 7f2359b8c700 1 --2- 172.21.15.103:0/3922952242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354133360 0x7f235409cc90 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.369+0000 7f2359b8c700 1 -- 172.21.15.103:0/3922952242 shutdown_connections 2023-07-02T08:59:48.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.369+0000 7f2359b8c700 1 --2- 172.21.15.103:0/3922952242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354133360 0x7f235409cc90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.369+0000 7f2359b8c700 1 -- 172.21.15.103:0/3922952242 >> 172.21.15.103:0/3922952242 conn(0x7f23540a2080 msgr2=0x7f23540a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:48.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.370+0000 7f2359b8c700 1 -- 172.21.15.103:0/3922952242 shutdown_connections 2023-07-02T08:59:48.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.370+0000 7f2359b8c700 1 -- 172.21.15.103:0/3922952242 wait complete. 2023-07-02T08:59:48.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.370+0000 7f2359b8c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:48.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.371+0000 7f2359b8c700 1 Processor -- start 2023-07-02T08:59:48.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.371+0000 7f2359b8c700 1 -- start start 2023-07-02T08:59:48.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.372+0000 7f2359b8c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354132f50 0x7f235409b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.372+0000 7f2359b8c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235412be10 con 0x7f2354132f50 2023-07-02T08:59:48.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.372+0000 7f23537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354132f50 0x7f235409b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.372+0000 7f23537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354132f50 0x7f235409b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36822/0 (socket says 172.21.15.103:36822) 2023-07-02T08:59:48.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.372+0000 7f23537fe700 1 -- 172.21.15.103:0/3232654380 learned_addr learned my addr 172.21.15.103:0/3232654380 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.372+0000 7f23537fe700 1 -- 172.21.15.103:0/3232654380 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235412ba70 con 0x7f2354132f50 2023-07-02T08:59:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.372+0000 7f23537fe700 1 --2- 172.21.15.103:0/3232654380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354132f50 0x7f235409b7c0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.373+0000 7f233bfff700 1 -- 172.21.15.103:0/3232654380 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f23440059a0 con 0x7f2354132f50 2023-07-02T08:59:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.373+0000 7f233bfff700 1 -- 172.21.15.103:0/3232654380 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2344005b00 con 0x7f2354132f50 2023-07-02T08:59:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.373+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235406fa20 con 0x7f2354132f50 2023-07-02T08:59:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.373+0000 7f233bfff700 1 -- 172.21.15.103:0/3232654380 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2344005da0 con 0x7f2354132f50 2023-07-02T08:59:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.373+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235406efc0 con 0x7f2354132f50 2023-07-02T08:59:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.374+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235412fda0 con 0x7f2354132f50 2023-07-02T08:59:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.374+0000 7f233bfff700 1 -- 172.21.15.103:0/3232654380 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f23440070b0 con 0x7f2354132f50 2023-07-02T08:59:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.374+0000 7f233bfff700 1 --2- 172.21.15.103:0/3232654380 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23400377c0 0x7f2340039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.374+0000 7f233bfff700 1 -- 172.21.15.103:0/3232654380 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f23440469e0 con 0x7f2354132f50 2023-07-02T08:59:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.375+0000 7f2352ffd700 1 --2- 172.21.15.103:0/3232654380 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23400377c0 0x7f2340039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.375+0000 7f2352ffd700 1 --2- 172.21.15.103:0/3232654380 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23400377c0 0x7f2340039cd0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.383+0000 7f233bfff700 1 -- 172.21.15.103:0/3232654380 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f234400a710 con 0x7f2354132f50 2023-07-02T08:59:48.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.403+0000 7f43d435f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:48.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.412+0000 7f43d435f700 1 Processor -- start 2023-07-02T08:59:48.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.412+0000 7f43d435f700 1 -- start start 2023-07-02T08:59:48.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d435f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f43cc133360 0x7f43cc09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d435f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc09e260 con 0x7f43cc132f50 2023-07-02T08:59:48.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d435f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43cc0a1640 con 0x7f43cc133360 2023-07-02T08:59:48.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d18fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f43cc133360 0x7f43cc09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d20fb700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f43cc132f50 0x7f43cc09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55648/0 (socket says 172.21.15.103:55648) 2023-07-02T08:59:48.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d20fb700 1 -- 172.21.15.103:0/150289213 learned_addr learned my addr 172.21.15.103:0/150289213 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d18fa700 1 -- 172.21.15.103:0/150289213 >> v1:172.21.15.103:6789/0 conn(0x7f43cc132f50 legacy=0x7f43cc09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d18fa700 1 -- 172.21.15.103:0/150289213 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc132a20 con 0x7f43cc133360 2023-07-02T08:59:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d10f9700 1 -- 172.21.15.103:0/150289213 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572524785 0 0) 0x7f43cc09e260 con 0x7f43cc132f50 2023-07-02T08:59:48.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.414+0000 7f43d18fa700 1 --2- 172.21.15.103:0/150289213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc133360 0x7f43cc09cc90 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f9e64def19360a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.415+0000 7f43d10f9700 1 -- 172.21.15.103:0/150289213 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f43bc009cd0 con 0x7f43cc133360 2023-07-02T08:59:48.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.415+0000 7f43d10f9700 1 -- 172.21.15.103:0/150289213 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f43bc009e30 con 0x7f43cc133360 2023-07-02T08:59:48.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.415+0000 7f43d10f9700 1 -- 172.21.15.103:0/150289213 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f43bc00e360 con 0x7f43cc133360 2023-07-02T08:59:48.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.415+0000 7f43d435f700 1 -- 172.21.15.103:0/150289213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc133360 msgr2=0x7f43cc09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.415+0000 7f43d435f700 1 --2- 172.21.15.103:0/150289213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc133360 0x7f43cc09cc90 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.416+0000 7f43d435f700 1 -- 172.21.15.103:0/150289213 shutdown_connections 2023-07-02T08:59:48.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.416+0000 7f43d435f700 1 --2- 172.21.15.103:0/150289213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc133360 0x7f43cc09cc90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.416+0000 7f43d435f700 1 -- 172.21.15.103:0/150289213 >> 172.21.15.103:0/150289213 conn(0x7f43cc0a2080 msgr2=0x7f43cc0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:48.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.416+0000 7f43d435f700 1 -- 172.21.15.103:0/150289213 shutdown_connections 2023-07-02T08:59:48.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.416+0000 7f43d435f700 1 -- 172.21.15.103:0/150289213 wait complete. 2023-07-02T08:59:48.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.416+0000 7f43d435f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:48.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.409+0000 7fe3d94f3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:48.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.417+0000 7fe3d94f3700 1 Processor -- start 2023-07-02T08:59:48.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.417+0000 7f43d435f700 1 Processor -- start 2023-07-02T08:59:48.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.417+0000 7f43d435f700 1 -- start start 2023-07-02T08:59:48.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.417+0000 7fe3d94f3700 1 -- start start 2023-07-02T08:59:48.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d94f3700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe3d4099520 0x7fe3d4134800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d94f3700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d409b330 con 0x7fe3d4098a00 2023-07-02T08:59:48.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d94f3700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d406d070 con 0x7fe3d4099520 2023-07-02T08:59:48.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe3d4099520 0x7fe3d4134800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe3d4099520 0x7fe3d4134800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36836/0 (socket says 172.21.15.103:36836) 2023-07-02T08:59:48.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d27fc700 1 -- 172.21.15.103:0/3741960770 learned_addr learned my addr 172.21.15.103:0/3741960770 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:48.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7f43d435f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc132f50 0x7f43cc12c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7f43d435f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc06d2f0 con 0x7f43cc132f50 2023-07-02T08:59:48.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d27fc700 1 -- 172.21.15.103:0/3741960770 >> v1:172.21.15.103:6789/0 conn(0x7fe3d4098a00 legacy=0x7fe3d4098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d27fc700 1 -- 172.21.15.103:0/3741960770 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d406d420 con 0x7fe3d4099520 2023-07-02T08:59:48.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7f43d20fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc132f50 0x7f43cc12c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7f43d20fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc132f50 0x7f43cc12c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36850/0 (socket says 172.21.15.103:36850) 2023-07-02T08:59:48.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7f43d20fb700 1 -- 172.21.15.103:0/978603275 learned_addr learned my addr 172.21.15.103:0/978603275 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:48.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.418+0000 7fe3d27fc700 1 --2- 172.21.15.103:0/3741960770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4099520 0x7fe3d4134800 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318e7175b73ca298 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7f43d20fb700 1 -- 172.21.15.103:0/978603275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc06dd60 con 0x7f43cc132f50 2023-07-02T08:59:48.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7f43d20fb700 1 --2- 172.21.15.103:0/978603275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc132f50 0x7f43cc12c1a0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7fe3d1ffb700 1 -- 172.21.15.103:0/3741960770 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe3c8010570 con 0x7fe3d4099520 2023-07-02T08:59:48.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7fe3d1ffb700 1 -- 172.21.15.103:0/3741960770 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe3c80106d0 con 0x7fe3d4099520 2023-07-02T08:59:48.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7fe3d1ffb700 1 -- 172.21.15.103:0/3741960770 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe3c8010970 con 0x7fe3d4099520 2023-07-02T08:59:48.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7f43c2ffd700 1 -- 172.21.15.103:0/978603275 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f43c8004b80 con 0x7f43cc132f50 2023-07-02T08:59:48.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7f43c2ffd700 1 -- 172.21.15.103:0/978603275 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f43c8004ce0 con 0x7f43cc132f50 2023-07-02T08:59:48.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7f43c2ffd700 1 -- 172.21.15.103:0/978603275 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f43c800c4f0 con 0x7f43cc132f50 2023-07-02T08:59:48.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43cc12bc50 con 0x7f43cc132f50 2023-07-02T08:59:48.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.419+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc06f850 con 0x7f43cc132f50 2023-07-02T08:59:48.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.420+0000 7fe3d94f3700 1 -- 172.21.15.103:0/3741960770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4099520 msgr2=0x7fe3d4134800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.420+0000 7fe3d94f3700 1 --2- 172.21.15.103:0/3741960770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4099520 0x7fe3d4134800 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.420+0000 7fe3d94f3700 1 -- 172.21.15.103:0/3741960770 shutdown_connections 2023-07-02T08:59:48.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.420+0000 7fe3d94f3700 1 --2- 172.21.15.103:0/3741960770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4099520 0x7fe3d4134800 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.420+0000 7fe3d94f3700 1 -- 172.21.15.103:0/3741960770 >> 172.21.15.103:0/3741960770 conn(0x7fe3d409f480 msgr2=0x7fe3d409c720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:48.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.420+0000 7fe3d94f3700 1 -- 172.21.15.103:0/3741960770 shutdown_connections 2023-07-02T08:59:48.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.420+0000 7fe3d94f3700 1 -- 172.21.15.103:0/3741960770 wait complete. 2023-07-02T08:59:48.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.421+0000 7fe3d94f3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:48.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.420+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc06dbb0 con 0x7f43cc132f50 2023-07-02T08:59:48.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.421+0000 7f43c2ffd700 1 -- 172.21.15.103:0/978603275 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f43c800c650 con 0x7f43cc132f50 2023-07-02T08:59:48.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.421+0000 7f43c2ffd700 1 --2- 172.21.15.103:0/978603275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43b00377c0 0x7f43b0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.422+0000 7fe3d94f3700 1 Processor -- start 2023-07-02T08:59:48.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.422+0000 7fe3d94f3700 1 -- start start 2023-07-02T08:59:48.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.423+0000 7fe3d94f3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4098a00 0x7fe3d41a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.423+0000 7fe3d94f3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d409a080 con 0x7fe3d4098a00 2023-07-02T08:59:48.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.424+0000 7fe3d2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4098a00 0x7fe3d41a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.424+0000 7fe3d2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4098a00 0x7fe3d41a1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36852/0 (socket says 172.21.15.103:36852) 2023-07-02T08:59:48.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.424+0000 7fe3d2ffd700 1 -- 172.21.15.103:0/2126456911 learned_addr learned my addr 172.21.15.103:0/2126456911 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:48.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.424+0000 7fe3d2ffd700 1 -- 172.21.15.103:0/2126456911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d406d730 con 0x7fe3d4098a00 2023-07-02T08:59:48.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.424+0000 7fe3d2ffd700 1 --2- 172.21.15.103:0/2126456911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4098a00 0x7fe3d41a1bd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.425+0000 7f43c2ffd700 1 -- 172.21.15.103:0/978603275 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f43c8009e80 con 0x7f43cc132f50 2023-07-02T08:59:48.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.425+0000 7f43d18fa700 1 --2- 172.21.15.103:0/978603275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43b00377c0 0x7f43b0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.425+0000 7f43d18fa700 1 --2- 172.21.15.103:0/978603275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43b00377c0 0x7f43b0039cd0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.430+0000 7fe3bb7fe700 1 -- 172.21.15.103:0/2126456911 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe3c400c8a0 con 0x7fe3d4098a00 2023-07-02T08:59:48.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.430+0000 7f43c2ffd700 1 -- 172.21.15.103:0/978603275 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f43c8009c80 con 0x7f43cc132f50 2023-07-02T08:59:48.435 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.430+0000 7fe3bb7fe700 1 -- 172.21.15.103:0/2126456911 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe3c400ca00 con 0x7fe3d4098a00 2023-07-02T08:59:48.435 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.430+0000 7fe3bb7fe700 1 -- 172.21.15.103:0/2126456911 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe3c400cca0 con 0x7fe3d4098a00 2023-07-02T08:59:48.435 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.430+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d41a2ed0 con 0x7fe3d4098a00 2023-07-02T08:59:48.435 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.430+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d41a41f0 con 0x7fe3d4098a00 2023-07-02T08:59:48.435 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.431+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d406cbe0 con 0x7fe3d4098a00 2023-07-02T08:59:48.435 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.431+0000 7fe3bb7fe700 1 -- 172.21.15.103:0/2126456911 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fe3c4011310 con 0x7fe3d4098a00 2023-07-02T08:59:48.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.432+0000 7fe3bb7fe700 1 --2- 172.21.15.103:0/2126456911 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3c00377c0 0x7fe3c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.432+0000 7fe3bb7fe700 1 -- 172.21.15.103:0/2126456911 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fe3c4047210 con 0x7fe3d4098a00 2023-07-02T08:59:48.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.432+0000 7fe3d27fc700 1 --2- 172.21.15.103:0/2126456911 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3c00377c0 0x7fe3c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.432+0000 7fe3d27fc700 1 --2- 172.21.15.103:0/2126456911 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3c00377c0 0x7fe3c0039cd0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.440+0000 7fe3bb7fe700 1 -- 172.21.15.103:0/2126456911 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fe3c400dd50 con 0x7fe3d4098a00 2023-07-02T08:59:48.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.454+0000 7efcfafd0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:48.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.459+0000 7efcfafd0700 1 Processor -- start 2023-07-02T08:59:48.461 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.460+0000 7efcfafd0700 1 -- start start 2023-07-02T08:59:48.461 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.460+0000 7efcfafd0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efcf4133360 0x7efcf409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.460+0000 7efcfafd0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf409e260 con 0x7efcf4132f50 2023-07-02T08:59:48.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.460+0000 7efcfafd0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf40a1640 con 0x7efcf4133360 2023-07-02T08:59:48.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.460+0000 7efcf3fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efcf4133360 0x7efcf409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.460+0000 7efcf8d6c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7efcf4132f50 0x7efcf409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55662/0 (socket says 172.21.15.103:55662) 2023-07-02T08:59:48.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.460+0000 7efcf8d6c700 1 -- 172.21.15.103:0/3664818651 learned_addr learned my addr 172.21.15.103:0/3664818651 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:48.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.461+0000 7efcf37fe700 1 -- 172.21.15.103:0/3664818651 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601768688 0 0) 0x7efcf409e260 con 0x7efcf4132f50 2023-07-02T08:59:48.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.461+0000 7efcf37fe700 1 -- 172.21.15.103:0/3664818651 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce4003580 con 0x7efcf4132f50 2023-07-02T08:59:48.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.461+0000 7efcf3fff700 1 -- 172.21.15.103:0/3664818651 >> v1:172.21.15.103:6789/0 conn(0x7efcf4132f50 legacy=0x7efcf409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.461+0000 7efcf3fff700 1 -- 172.21.15.103:0/3664818651 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf4132a20 con 0x7efcf4133360 2023-07-02T08:59:48.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.461+0000 7efcf3fff700 1 --2- 172.21.15.103:0/3664818651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4133360 0x7efcf409cc90 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f5e25843ea8169 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.461+0000 7efcf37fe700 1 -- 172.21.15.103:0/3664818651 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efce0009c80 con 0x7efcf4133360 2023-07-02T08:59:48.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.461+0000 7efcf37fe700 1 -- 172.21.15.103:0/3664818651 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efce0009de0 con 0x7efcf4133360 2023-07-02T08:59:48.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.461+0000 7efcf37fe700 1 -- 172.21.15.103:0/3664818651 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efce000e360 con 0x7efcf4133360 2023-07-02T08:59:48.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.462+0000 7efcfafd0700 1 -- 172.21.15.103:0/3664818651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4133360 msgr2=0x7efcf409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.462+0000 7efcfafd0700 1 --2- 172.21.15.103:0/3664818651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4133360 0x7efcf409cc90 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.462+0000 7efcfafd0700 1 -- 172.21.15.103:0/3664818651 shutdown_connections 2023-07-02T08:59:48.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.462+0000 7efcfafd0700 1 --2- 172.21.15.103:0/3664818651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4133360 0x7efcf409cc90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.462+0000 7efcfafd0700 1 -- 172.21.15.103:0/3664818651 >> 172.21.15.103:0/3664818651 conn(0x7efcf40a2080 msgr2=0x7efcf40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:48.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.462+0000 7efcfafd0700 1 -- 172.21.15.103:0/3664818651 shutdown_connections 2023-07-02T08:59:48.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.462+0000 7efcfafd0700 1 -- 172.21.15.103:0/3664818651 wait complete. 2023-07-02T08:59:48.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.462+0000 7efcfafd0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:48.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.463+0000 7efcfafd0700 1 Processor -- start 2023-07-02T08:59:48.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.463+0000 7efcfafd0700 1 -- start start 2023-07-02T08:59:48.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.464+0000 7efcfafd0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4132f50 0x7efcf412b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.464+0000 7efcfafd0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf406ec20 con 0x7efcf4132f50 2023-07-02T08:59:48.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.464+0000 7efcf8d6c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4132f50 0x7efcf412b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.464+0000 7efcf8d6c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4132f50 0x7efcf412b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36856/0 (socket says 172.21.15.103:36856) 2023-07-02T08:59:48.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.464+0000 7efcf8d6c700 1 -- 172.21.15.103:0/3286464852 learned_addr learned my addr 172.21.15.103:0/3286464852 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:48.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.464+0000 7efcf8d6c700 1 -- 172.21.15.103:0/3286464852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf406fa10 con 0x7efcf4132f50 2023-07-02T08:59:48.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.464+0000 7efcf8d6c700 1 --2- 172.21.15.103:0/3286464852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4132f50 0x7efcf412b790 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.465+0000 7efcf17fa700 1 -- 172.21.15.103:0/3286464852 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efce8005a60 con 0x7efcf4132f50 2023-07-02T08:59:48.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.465+0000 7efcf17fa700 1 -- 172.21.15.103:0/3286464852 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efce8005bc0 con 0x7efcf4132f50 2023-07-02T08:59:48.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.465+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf406dba0 con 0x7efcf4132f50 2023-07-02T08:59:48.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.465+0000 7efcf17fa700 1 -- 172.21.15.103:0/3286464852 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efce8005e60 con 0x7efcf4132f50 2023-07-02T08:59:48.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.465+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf406f700 con 0x7efcf4132f50 2023-07-02T08:59:48.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.466+0000 7efcf17fa700 1 -- 172.21.15.103:0/3286464852 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7efce80070b0 con 0x7efcf4132f50 2023-07-02T08:59:48.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.466+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf4061ea0 con 0x7efcf4132f50 2023-07-02T08:59:48.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.466+0000 7efcf17fa700 1 --2- 172.21.15.103:0/3286464852 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efcd8037810 0x7efcd8039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:48.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.466+0000 7efcf17fa700 1 -- 172.21.15.103:0/3286464852 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7efce8046ba0 con 0x7efcf4132f50 2023-07-02T08:59:48.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.466+0000 7efcf3fff700 1 --2- 172.21.15.103:0/3286464852 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efcd8037810 0x7efcd8039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:48.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.467+0000 7efcf3fff700 1 --2- 172.21.15.103:0/3286464852 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efcd8037810 0x7efcd8039d20 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:48.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.471+0000 7efcf17fa700 1 -- 172.21.15.103:0/3286464852 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7efce800a4e0 con 0x7efcf4132f50 2023-07-02T08:59:48.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.621+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f235406efc0 con 0x7f2354132f50 2023-07-02T08:59:48.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.622+0000 7f233bfff700 1 -- 172.21.15.103:0/3232654380 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f235406efc0 con 0x7f2354132f50 2023-07-02T08:59:48.623 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:48.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23400377c0 msgr2=0x7f2340039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 --2- 172.21.15.103:0/3232654380 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23400377c0 0x7f2340039cd0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354132f50 msgr2=0x7f235409b7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 --2- 172.21.15.103:0/3232654380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354132f50 0x7f235409b7c0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 shutdown_connections 2023-07-02T08:59:48.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 --2- 172.21.15.103:0/3232654380 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23400377c0 0x7f2340039cd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 --2- 172.21.15.103:0/3232654380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2354132f50 0x7f235409b7c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 >> 172.21.15.103:0/3232654380 conn(0x7f23540a2080 msgr2=0x7f23541bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:48.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 shutdown_connections 2023-07-02T08:59:48.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.624+0000 7f2359b8c700 1 -- 172.21.15.103:0/3232654380 wait complete. 2023-07-02T08:59:48.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.629+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f43cc061ea0 con 0x7f43cc132f50 2023-07-02T08:59:48.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.633+0000 7f43c2ffd700 1 -- 172.21.15.103:0/978603275 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f43c80098a0 con 0x7f43cc132f50 2023-07-02T08:59:48.634 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43b00377c0 msgr2=0x7f43b0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 --2- 172.21.15.103:0/978603275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43b00377c0 0x7f43b0039cd0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc132f50 msgr2=0x7f43cc12c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 --2- 172.21.15.103:0/978603275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc132f50 0x7f43cc12c1a0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 shutdown_connections 2023-07-02T08:59:48.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 --2- 172.21.15.103:0/978603275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43b00377c0 0x7f43b0039cd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 --2- 172.21.15.103:0/978603275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43cc132f50 0x7f43cc12c1a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 >> 172.21.15.103:0/978603275 conn(0x7f43cc0a2080 msgr2=0x7f43cc130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:48.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 shutdown_connections 2023-07-02T08:59:48.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.635+0000 7f43d435f700 1 -- 172.21.15.103:0/978603275 wait complete. 2023-07-02T08:59:48.640 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:48.650 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:48.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.662+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe3d4061ea0 con 0x7fe3d4098a00 2023-07-02T08:59:48.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.662+0000 7fe3bb7fe700 1 -- 172.21.15.103:0/2126456911 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fe3c400d940 con 0x7fe3d4098a00 2023-07-02T08:59:48.664 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:48.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3c00377c0 msgr2=0x7fe3c0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 --2- 172.21.15.103:0/2126456911 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3c00377c0 0x7fe3c0039cd0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4098a00 msgr2=0x7fe3d41a1bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 --2- 172.21.15.103:0/2126456911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4098a00 0x7fe3d41a1bd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 shutdown_connections 2023-07-02T08:59:48.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 --2- 172.21.15.103:0/2126456911 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3c00377c0 0x7fe3c0039cd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 --2- 172.21.15.103:0/2126456911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3d4098a00 0x7fe3d41a1bd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 >> 172.21.15.103:0/2126456911 conn(0x7fe3d409f480 msgr2=0x7fe3d41b6420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:48.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 shutdown_connections 2023-07-02T08:59:48.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.665+0000 7fe3d94f3700 1 -- 172.21.15.103:0/2126456911 wait complete. 2023-07-02T08:59:48.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.668+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7efcf40725b0 con 0x7efcf4132f50 2023-07-02T08:59:48.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.668+0000 7efcf17fa700 1 -- 172.21.15.103:0/3286464852 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7efce800a120 con 0x7efcf4132f50 2023-07-02T08:59:48.670 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:48.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efcd8037810 msgr2=0x7efcd8039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 --2- 172.21.15.103:0/3286464852 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efcd8037810 0x7efcd8039d20 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4132f50 msgr2=0x7efcf412b790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:48.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 --2- 172.21.15.103:0/3286464852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4132f50 0x7efcf412b790 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 shutdown_connections 2023-07-02T08:59:48.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 --2- 172.21.15.103:0/3286464852 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efcd8037810 0x7efcd8039d20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 --2- 172.21.15.103:0/3286464852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcf4132f50 0x7efcf412b790 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:48.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 >> 172.21.15.103:0/3286464852 conn(0x7efcf40a2080 msgr2=0x7efcf41bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:48.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 shutdown_connections 2023-07-02T08:59:48.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:48.670+0000 7efcfafd0700 1 -- 172.21.15.103:0/3286464852 wait complete. 2023-07-02T08:59:48.681 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:48.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:49.641 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:49.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:49.682 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:49.687 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:49.806 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.795+0000 7f682db52700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:49.807 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.806+0000 7f682db52700 1 Processor -- start 2023-07-02T08:59:49.807 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.806+0000 7f682db52700 1 -- start start 2023-07-02T08:59:49.808 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.807+0000 7f682db52700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f682809d8a0 0x7f682809dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.808 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.807+0000 7f682db52700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682809b590 con 0x7f682809e1f0 2023-07-02T08:59:49.808 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.807+0000 7f682db52700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682809a5b0 con 0x7f682809d8a0 2023-07-02T08:59:49.809 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.807+0000 7f68277fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f682809d8a0 0x7f682809dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.809 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.807+0000 7f68277fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f682809d8a0 0x7f682809dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36872/0 (socket says 172.21.15.103:36872) 2023-07-02T08:59:49.809 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.807+0000 7f68277fe700 1 -- 172.21.15.103:0/2897311185 learned_addr learned my addr 172.21.15.103:0/2897311185 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:49.810 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.807+0000 7f68277fe700 1 -- 172.21.15.103:0/2897311185 >> v1:172.21.15.103:6789/0 conn(0x7f682809e1f0 legacy=0x7f6828130660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:49.810 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.807+0000 7f68277fe700 1 -- 172.21.15.103:0/2897311185 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682809f700 con 0x7f682809d8a0 2023-07-02T08:59:49.810 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.808+0000 7f68277fe700 1 --2- 172.21.15.103:0/2897311185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f682809dcb0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7812b131c61591dd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.810 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.808+0000 7f68267fc700 1 -- 172.21.15.103:0/2897311185 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6818009d90 con 0x7f682809d8a0 2023-07-02T08:59:49.810 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.808+0000 7f68267fc700 1 -- 172.21.15.103:0/2897311185 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f681800e230 con 0x7f682809d8a0 2023-07-02T08:59:49.810 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.808+0000 7f68267fc700 1 -- 172.21.15.103:0/2897311185 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f681800e4d0 con 0x7f682809d8a0 2023-07-02T08:59:49.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.809+0000 7f682db52700 1 -- 172.21.15.103:0/2897311185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 msgr2=0x7f682809dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:49.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.809+0000 7f682db52700 1 --2- 172.21.15.103:0/2897311185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f682809dcb0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:49.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.809+0000 7f682db52700 1 -- 172.21.15.103:0/2897311185 shutdown_connections 2023-07-02T08:59:49.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.809+0000 7f682db52700 1 --2- 172.21.15.103:0/2897311185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f682809dcb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:49.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.809+0000 7f682db52700 1 -- 172.21.15.103:0/2897311185 >> 172.21.15.103:0/2897311185 conn(0x7f68280a3280 msgr2=0x7f68280a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:49.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.809+0000 7f682db52700 1 -- 172.21.15.103:0/2897311185 shutdown_connections 2023-07-02T08:59:49.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.809+0000 7f682db52700 1 -- 172.21.15.103:0/2897311185 wait complete. 2023-07-02T08:59:49.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.809+0000 7f682db52700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:49.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.811+0000 7f682db52700 1 Processor -- start 2023-07-02T08:59:49.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.811+0000 7f682db52700 1 -- start start 2023-07-02T08:59:49.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.812+0000 7f682db52700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f68281c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.812+0000 7f682db52700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682806e570 con 0x7f682809d8a0 2023-07-02T08:59:49.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.812+0000 7f68277fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f68281c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.812+0000 7f68277fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f68281c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36878/0 (socket says 172.21.15.103:36878) 2023-07-02T08:59:49.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.812+0000 7f68277fe700 1 -- 172.21.15.103:0/230035632 learned_addr learned my addr 172.21.15.103:0/230035632 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:49.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.812+0000 7f68277fe700 1 -- 172.21.15.103:0/230035632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68281c6c10 con 0x7f682809d8a0 2023-07-02T08:59:49.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.813+0000 7f68277fe700 1 --2- 172.21.15.103:0/230035632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f68281c7250 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.813+0000 7f680ffff700 1 -- 172.21.15.103:0/230035632 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6818014540 con 0x7f682809d8a0 2023-07-02T08:59:49.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.813+0000 7f680ffff700 1 -- 172.21.15.103:0/230035632 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f68180146a0 con 0x7f682809d8a0 2023-07-02T08:59:49.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.813+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68281c6eb0 con 0x7f682809d8a0 2023-07-02T08:59:49.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.813+0000 7f680ffff700 1 -- 172.21.15.103:0/230035632 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6818014940 con 0x7f682809d8a0 2023-07-02T08:59:49.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.813+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68281a9100 con 0x7f682809d8a0 2023-07-02T08:59:49.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.814+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682809b250 con 0x7f682809d8a0 2023-07-02T08:59:49.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.814+0000 7f680ffff700 1 -- 172.21.15.103:0/230035632 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6818014aa0 con 0x7f682809d8a0 2023-07-02T08:59:49.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.815+0000 7f680ffff700 1 --2- 172.21.15.103:0/230035632 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6814037810 0x7f6814039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.815+0000 7f680ffff700 1 -- 172.21.15.103:0/230035632 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6818019040 con 0x7f682809d8a0 2023-07-02T08:59:49.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.815+0000 7f6826ffd700 1 --2- 172.21.15.103:0/230035632 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6814037810 0x7f6814039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.817+0000 7f6826ffd700 1 --2- 172.21.15.103:0/230035632 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6814037810 0x7f6814039d20 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.819+0000 7f680ffff700 1 -- 172.21.15.103:0/230035632 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6818018190 con 0x7f682809d8a0 2023-07-02T08:59:49.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.828+0000 7f232b111700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:49.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.828+0000 7f485bcea700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:49.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.834+0000 7f232b111700 1 Processor -- start 2023-07-02T08:59:49.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.834+0000 7f485bcea700 1 Processor -- start 2023-07-02T08:59:49.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.834+0000 7f232b111700 1 -- start start 2023-07-02T08:59:49.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.834+0000 7f485bcea700 1 -- start start 2023-07-02T08:59:49.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f232b111700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2324132f50 0x7f2324133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f232b111700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232409e260 con 0x7f232409c580 2023-07-02T08:59:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f232b111700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23240a1640 con 0x7f2324132f50 2023-07-02T08:59:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f2328ead700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2324132f50 0x7f2324133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f2328ead700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2324132f50 0x7f2324133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36892/0 (socket says 172.21.15.103:36892) 2023-07-02T08:59:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f2328ead700 1 -- 172.21.15.103:0/102754966 learned_addr learned my addr 172.21.15.103:0/102754966 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:49.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f485bcea700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f485409e3c0 0x7f48541358a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f485bcea700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485409b590 con 0x7f485409d8a0 2023-07-02T08:59:49.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f485bcea700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485409a5b0 con 0x7f485409e3c0 2023-07-02T08:59:49.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f4859285700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f485409e3c0 0x7f48541358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f4859a86700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f485409d8a0 0x7f485409dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55686/0 (socket says 172.21.15.103:55686) 2023-07-02T08:59:49.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f4859a86700 1 -- 172.21.15.103:0/107412861 learned_addr learned my addr 172.21.15.103:0/107412861 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f2328ead700 1 -- 172.21.15.103:0/102754966 >> v1:172.21.15.103:6789/0 conn(0x7f232409c580 legacy=0x7f232409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-02T08:59:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f2328ead700 1 -- 172.21.15.103:0/102754966 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2324132a20 con 0x7f2324132f50 2023-07-02T08:59:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f4859285700 1 -- 172.21.15.103:0/107412861 >> v1:172.21.15.103:6789/0 conn(0x7f485409d8a0 legacy=0x7f485409dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f4859285700 1 -- 172.21.15.103:0/107412861 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485409f700 con 0x7f485409e3c0 2023-07-02T08:59:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.836+0000 7f4859285700 1 --2- 172.21.15.103:0/107412861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409e3c0 0x7f48541358a0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d309563347d2ae59 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.835+0000 7f2328ead700 1 --2- 172.21.15.103:0/102754966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2324132f50 0x7f2324133360 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d988a937d0427a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.837+0000 7f4858a84700 1 -- 172.21.15.103:0/107412861 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4850009cd0 con 0x7f485409e3c0 2023-07-02T08:59:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.837+0000 7f4858a84700 1 -- 172.21.15.103:0/107412861 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4850009e30 con 0x7f485409e3c0 2023-07-02T08:59:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.837+0000 7f4858a84700 1 -- 172.21.15.103:0/107412861 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f485000e360 con 0x7f485409e3c0 2023-07-02T08:59:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.837+0000 7f23237fe700 1 -- 172.21.15.103:0/102754966 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2318009d00 con 0x7f2324132f50 2023-07-02T08:59:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.837+0000 7f23237fe700 1 -- 172.21.15.103:0/102754966 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2318009e60 con 0x7f2324132f50 2023-07-02T08:59:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.837+0000 7f23237fe700 1 -- 172.21.15.103:0/102754966 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f231800e360 con 0x7f2324132f50 2023-07-02T08:59:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f485bcea700 1 -- 172.21.15.103:0/107412861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409e3c0 msgr2=0x7f48541358a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f485bcea700 1 --2- 172.21.15.103:0/107412861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409e3c0 0x7f48541358a0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f232b111700 1 -- 172.21.15.103:0/102754966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2324132f50 msgr2=0x7f2324133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:49.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f232b111700 1 --2- 172.21.15.103:0/102754966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2324132f50 0x7f2324133360 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:49.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f232b111700 1 -- 172.21.15.103:0/102754966 shutdown_connections 2023-07-02T08:59:49.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f232b111700 1 --2- 172.21.15.103:0/102754966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2324132f50 0x7f2324133360 crc :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:49.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f232b111700 1 -- 172.21.15.103:0/102754966 >> 172.21.15.103:0/102754966 conn(0x7f23240a2080 msgr2=0x7f23240a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:49.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f485bcea700 1 -- 172.21.15.103:0/107412861 shutdown_connections 2023-07-02T08:59:49.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f485bcea700 1 --2- 172.21.15.103:0/107412861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409e3c0 0x7f48541358a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f485bcea700 1 -- 172.21.15.103:0/107412861 >> 172.21.15.103:0/107412861 conn(0x7f48540a3280 msgr2=0x7f48540a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f232b111700 1 -- 172.21.15.103:0/102754966 shutdown_connections 2023-07-02T08:59:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f485bcea700 1 -- 172.21.15.103:0/107412861 shutdown_connections 2023-07-02T08:59:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f485bcea700 1 -- 172.21.15.103:0/107412861 wait complete. 2023-07-02T08:59:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f232b111700 1 -- 172.21.15.103:0/102754966 wait complete. 2023-07-02T08:59:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f232b111700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.838+0000 7f485bcea700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.839+0000 7f485bcea700 1 Processor -- start 2023-07-02T08:59:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.839+0000 7f232b111700 1 Processor -- start 2023-07-02T08:59:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.839+0000 7f485bcea700 1 -- start start 2023-07-02T08:59:49.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.840+0000 7f485bcea700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409d8a0 0x7f48541cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.840+0000 7f485bcea700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48541b8260 con 0x7f485409d8a0 2023-07-02T08:59:49.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.839+0000 7f232b111700 1 -- start start 2023-07-02T08:59:49.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.840+0000 7f232b111700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f232409c580 0x7f232406d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.840+0000 7f232b111700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232412c0e0 con 0x7f232409c580 2023-07-02T08:59:49.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.840+0000 7f4859a86700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409d8a0 0x7f48541cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.840+0000 7f4859a86700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409d8a0 0x7f48541cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36906/0 (socket says 172.21.15.103:36906) 2023-07-02T08:59:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.840+0000 7f4859a86700 1 -- 172.21.15.103:0/3625360426 learned_addr learned my addr 172.21.15.103:0/3625360426 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.840+0000 7f4859a86700 1 -- 172.21.15.103:0/3625360426 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48541b8030 con 0x7f485409d8a0 2023-07-02T08:59:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f4859a86700 1 --2- 172.21.15.103:0/3625360426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409d8a0 0x7f48541cb930 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f484a7fc700 1 -- 172.21.15.103:0/3625360426 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4844005a60 con 0x7f485409d8a0 2023-07-02T08:59:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f484a7fc700 1 -- 172.21.15.103:0/3625360426 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4844005bc0 con 0x7f485409d8a0 2023-07-02T08:59:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f484a7fc700 1 -- 172.21.15.103:0/3625360426 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4844005e60 con 0x7f485409d8a0 2023-07-02T08:59:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48541af720 con 0x7f485409d8a0 2023-07-02T08:59:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48541a1980 con 0x7f485409d8a0 2023-07-02T08:59:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f2328ead700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f232409c580 0x7f232406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f2328ead700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f232409c580 0x7f232406d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36912/0 (socket says 172.21.15.103:36912) 2023-07-02T08:59:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.841+0000 7f2328ead700 1 -- 172.21.15.103:0/1316611867 learned_addr learned my addr 172.21.15.103:0/1316611867 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.842+0000 7f2328ead700 1 -- 172.21.15.103:0/1316611867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232406f330 con 0x7f232409c580 2023-07-02T08:59:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.842+0000 7f2328ead700 1 --2- 172.21.15.103:0/1316611867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f232409c580 0x7f232406d070 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.842+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485409eda0 con 0x7f485409d8a0 2023-07-02T08:59:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.842+0000 7f23217fa700 1 -- 172.21.15.103:0/1316611867 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2318014410 con 0x7f232409c580 2023-07-02T08:59:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.842+0000 7f23217fa700 1 -- 172.21.15.103:0/1316611867 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2318014570 con 0x7f232409c580 2023-07-02T08:59:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.842+0000 7f23217fa700 1 -- 172.21.15.103:0/1316611867 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2318014810 con 0x7f232409c580 2023-07-02T08:59:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.842+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232406fd70 con 0x7f232409c580 2023-07-02T08:59:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.842+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232412bd40 con 0x7f232409c580 2023-07-02T08:59:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.843+0000 7f484a7fc700 1 -- 172.21.15.103:0/3625360426 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f48440070b0 con 0x7f485409d8a0 2023-07-02T08:59:49.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.843+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232412c8f0 con 0x7f232409c580 2023-07-02T08:59:49.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.844+0000 7f484a7fc700 1 --2- 172.21.15.103:0/3625360426 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48380377c0 0x7f4838039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.844+0000 7f23217fa700 1 -- 172.21.15.103:0/1316611867 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f23180149c0 con 0x7f232409c580 2023-07-02T08:59:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.844+0000 7f484a7fc700 1 -- 172.21.15.103:0/3625360426 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4844046db0 con 0x7f485409d8a0 2023-07-02T08:59:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.844+0000 7f23217fa700 1 --2- 172.21.15.103:0/1316611867 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23080377c0 0x7f2308039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.844+0000 7f4859285700 1 --2- 172.21.15.103:0/3625360426 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48380377c0 0x7f4838039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.844+0000 7f2323fff700 1 --2- 172.21.15.103:0/1316611867 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23080377c0 0x7f2308039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.844+0000 7f4859285700 1 --2- 172.21.15.103:0/3625360426 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48380377c0 0x7f4838039cd0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.844+0000 7f2323fff700 1 --2- 172.21.15.103:0/1316611867 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23080377c0 0x7f2308039cd0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.852+0000 7f484a7fc700 1 -- 172.21.15.103:0/3625360426 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f484400d510 con 0x7f485409d8a0 2023-07-02T08:59:49.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.852+0000 7f23217fa700 1 -- 172.21.15.103:0/1316611867 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2318047450 con 0x7f232409c580 2023-07-02T08:59:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.862+0000 7f23217fa700 1 -- 172.21.15.103:0/1316611867 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f231800fb20 con 0x7f232409c580 2023-07-02T08:59:49.889 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.879+0000 7f95db9e3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:49.889 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.889+0000 7f95db9e3700 1 Processor -- start 2023-07-02T08:59:49.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.889+0000 7f95db9e3700 1 -- start start 2023-07-02T08:59:49.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.890+0000 7f95db9e3700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f95d4133360 0x7f95d409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.890+0000 7f95db9e3700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d409e260 con 0x7f95d4132f50 2023-07-02T08:59:49.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.890+0000 7f95db9e3700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d40a1640 con 0x7f95d4133360 2023-07-02T08:59:49.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.890+0000 7f95d8f7e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f95d4133360 0x7f95d409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.890+0000 7f95d8f7e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f95d4133360 0x7f95d409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36922/0 (socket says 172.21.15.103:36922) 2023-07-02T08:59:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.890+0000 7f95d977f700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f95d4132f50 0x7f95d409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55690/0 (socket says 172.21.15.103:55690) 2023-07-02T08:59:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.890+0000 7f95d8f7e700 1 -- 172.21.15.103:0/4284091729 learned_addr learned my addr 172.21.15.103:0/4284091729 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.891+0000 7f95d8f7e700 1 -- 172.21.15.103:0/4284091729 >> v1:172.21.15.103:6789/0 conn(0x7f95d4132f50 legacy=0x7f95d409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.891+0000 7f95d8f7e700 1 -- 172.21.15.103:0/4284091729 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d4132a20 con 0x7f95d4133360 2023-07-02T08:59:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.891+0000 7f95cbfff700 1 -- 172.21.15.103:0/4284091729 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824650139 0 0) 0x7f95d409e260 con 0x7f95d4132f50 2023-07-02T08:59:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.891+0000 7f95d8f7e700 1 --2- 172.21.15.103:0/4284091729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4133360 0x7f95d409cc90 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ce7429fd1fcda8 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.891+0000 7f95cbfff700 1 -- 172.21.15.103:0/4284091729 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f95c4009d90 con 0x7f95d4133360 2023-07-02T08:59:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.891+0000 7f95cbfff700 1 -- 172.21.15.103:0/4284091729 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f95c400e230 con 0x7f95d4133360 2023-07-02T08:59:49.894 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.891+0000 7f95cbfff700 1 -- 172.21.15.103:0/4284091729 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f95c400e4d0 con 0x7f95d4133360 2023-07-02T08:59:49.894 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.892+0000 7f95db9e3700 1 -- 172.21.15.103:0/4284091729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4133360 msgr2=0x7f95d409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:49.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.892+0000 7f95db9e3700 1 --2- 172.21.15.103:0/4284091729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4133360 0x7f95d409cc90 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:49.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.892+0000 7f95db9e3700 1 -- 172.21.15.103:0/4284091729 shutdown_connections 2023-07-02T08:59:49.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.892+0000 7f95db9e3700 1 --2- 172.21.15.103:0/4284091729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4133360 0x7f95d409cc90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:49.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.892+0000 7f95db9e3700 1 -- 172.21.15.103:0/4284091729 >> 172.21.15.103:0/4284091729 conn(0x7f95d40a2080 msgr2=0x7f95d40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:49.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.892+0000 7f95db9e3700 1 -- 172.21.15.103:0/4284091729 shutdown_connections 2023-07-02T08:59:49.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.892+0000 7f95db9e3700 1 -- 172.21.15.103:0/4284091729 wait complete. 2023-07-02T08:59:49.896 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.892+0000 7f95db9e3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:49.896 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.894+0000 7f95db9e3700 1 Processor -- start 2023-07-02T08:59:49.896 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.894+0000 7f95db9e3700 1 -- start start 2023-07-02T08:59:49.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.894+0000 7f95db9e3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4132f50 0x7f95d41ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.894+0000 7f95db9e3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d406d370 con 0x7f95d4132f50 2023-07-02T08:59:49.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95d977f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4132f50 0x7f95d41ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95d977f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4132f50 0x7f95d41ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36938/0 (socket says 172.21.15.103:36938) 2023-07-02T08:59:49.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95d977f700 1 -- 172.21.15.103:0/4015884927 learned_addr learned my addr 172.21.15.103:0/4015884927 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:49.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95d977f700 1 -- 172.21.15.103:0/4015884927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d406dde0 con 0x7f95d4132f50 2023-07-02T08:59:49.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95d977f700 1 --2- 172.21.15.103:0/4015884927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4132f50 0x7f95d41ba9b0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95c9ffb700 1 -- 172.21.15.103:0/4015884927 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f95d0004b80 con 0x7f95d4132f50 2023-07-02T08:59:49.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95c9ffb700 1 -- 172.21.15.103:0/4015884927 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f95d0004ce0 con 0x7f95d4132f50 2023-07-02T08:59:49.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d41ba4f0 con 0x7f95d4132f50 2023-07-02T08:59:49.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95c9ffb700 1 -- 172.21.15.103:0/4015884927 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f95d000c4f0 con 0x7f95d4132f50 2023-07-02T08:59:49.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.895+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d4071e40 con 0x7f95d4132f50 2023-07-02T08:59:49.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.897+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d4061ea0 con 0x7f95d4132f50 2023-07-02T08:59:49.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.897+0000 7f95c9ffb700 1 -- 172.21.15.103:0/4015884927 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f95d000c650 con 0x7f95d4132f50 2023-07-02T08:59:49.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.897+0000 7f95c9ffb700 1 --2- 172.21.15.103:0/4015884927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f95b80377c0 0x7f95b8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:49.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.897+0000 7f95c9ffb700 1 -- 172.21.15.103:0/4015884927 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f95d0046970 con 0x7f95d4132f50 2023-07-02T08:59:49.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.897+0000 7f95d8f7e700 1 --2- 172.21.15.103:0/4015884927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f95b80377c0 0x7f95b8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:49.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.898+0000 7f95d8f7e700 1 --2- 172.21.15.103:0/4015884927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f95b80377c0 0x7f95b8039cd0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:49.906+0000 7f95c9ffb700 1 -- 172.21.15.103:0/4015884927 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f95d0009c80 con 0x7f95d4132f50 2023-07-02T08:59:50.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.034+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6828061ea0 con 0x7f682809d8a0 2023-07-02T08:59:50.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.035+0000 7f680ffff700 1 -- 172.21.15.103:0/230035632 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f681800fa90 con 0x7f682809d8a0 2023-07-02T08:59:50.036 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:50.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.037+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6814037810 msgr2=0x7f6814039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:50.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.037+0000 7f682db52700 1 --2- 172.21.15.103:0/230035632 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6814037810 0x7f6814039d20 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.037+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 msgr2=0x7f68281c7250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:50.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.037+0000 7f682db52700 1 --2- 172.21.15.103:0/230035632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f68281c7250 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.038+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 shutdown_connections 2023-07-02T08:59:50.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.038+0000 7f682db52700 1 --2- 172.21.15.103:0/230035632 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6814037810 0x7f6814039d20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.038+0000 7f682db52700 1 --2- 172.21.15.103:0/230035632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682809d8a0 0x7f68281c7250 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.038+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 >> 172.21.15.103:0/230035632 conn(0x7f68280a3280 msgr2=0x7f682819e200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:50.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.038+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 shutdown_connections 2023-07-02T08:59:50.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.038+0000 7f682db52700 1 -- 172.21.15.103:0/230035632 wait complete. 2023-07-02T08:59:50.053 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:50.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.054+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f48541a0230 con 0x7f485409d8a0 2023-07-02T08:59:50.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.054+0000 7f484a7fc700 1 -- 172.21.15.103:0/3625360426 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f484400d150 con 0x7f485409d8a0 2023-07-02T08:59:50.056 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48380377c0 msgr2=0x7f4838039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 --2- 172.21.15.103:0/3625360426 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48380377c0 0x7f4838039cd0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409d8a0 msgr2=0x7f48541cb930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 --2- 172.21.15.103:0/3625360426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409d8a0 0x7f48541cb930 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 shutdown_connections 2023-07-02T08:59:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 --2- 172.21.15.103:0/3625360426 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48380377c0 0x7f4838039cd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 --2- 172.21.15.103:0/3625360426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485409d8a0 0x7f48541cb930 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 >> 172.21.15.103:0/3625360426 conn(0x7f48540a3280 msgr2=0x7f485406d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:50.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 shutdown_connections 2023-07-02T08:59:50.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.056+0000 7f485bcea700 1 -- 172.21.15.103:0/3625360426 wait complete. 2023-07-02T08:59:50.071 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:50.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.075+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f232409e830 con 0x7f232409c580 2023-07-02T08:59:50.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.075+0000 7f23217fa700 1 -- 172.21.15.103:0/1316611867 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f231800f710 con 0x7f232409c580 2023-07-02T08:59:50.077 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:50.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.077+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23080377c0 msgr2=0x7f2308039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:50.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.077+0000 7f232b111700 1 --2- 172.21.15.103:0/1316611867 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23080377c0 0x7f2308039cd0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.077+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f232409c580 msgr2=0x7f232406d070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:50.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.077+0000 7f232b111700 1 --2- 172.21.15.103:0/1316611867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f232409c580 0x7f232406d070 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.078+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 shutdown_connections 2023-07-02T08:59:50.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.078+0000 7f232b111700 1 --2- 172.21.15.103:0/1316611867 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23080377c0 0x7f2308039cd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.078+0000 7f232b111700 1 --2- 172.21.15.103:0/1316611867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f232409c580 0x7f232406d070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.078+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 >> 172.21.15.103:0/1316611867 conn(0x7f23240a2080 msgr2=0x7f23241babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:50.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.078+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 shutdown_connections 2023-07-02T08:59:50.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.078+0000 7f232b111700 1 -- 172.21.15.103:0/1316611867 wait complete. 2023-07-02T08:59:50.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.105+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f95d40725b0 con 0x7f95d4132f50 2023-07-02T08:59:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.105+0000 7f95c9ffb700 1 -- 172.21.15.103:0/4015884927 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f95d00098a0 con 0x7f95d4132f50 2023-07-02T08:59:50.107 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:50.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f95b80377c0 msgr2=0x7f95b8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:50.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 --2- 172.21.15.103:0/4015884927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f95b80377c0 0x7f95b8039cd0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4132f50 msgr2=0x7f95d41ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:50.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 --2- 172.21.15.103:0/4015884927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4132f50 0x7f95d41ba9b0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 shutdown_connections 2023-07-02T08:59:50.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 --2- 172.21.15.103:0/4015884927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f95b80377c0 0x7f95b8039cd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 --2- 172.21.15.103:0/4015884927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95d4132f50 0x7f95d41ba9b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:50.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 >> 172.21.15.103:0/4015884927 conn(0x7f95d40a2080 msgr2=0x7f95d40a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:50.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 shutdown_connections 2023-07-02T08:59:50.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:50.108+0000 7f95db9e3700 1 -- 172.21.15.103:0/4015884927 wait complete. 2023-07-02T08:59:50.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:51.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:51.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:51.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:51.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:51.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.201+0000 7f8b6a267700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:51.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.208+0000 7f8b6a267700 1 Processor -- start 2023-07-02T08:59:51.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.208+0000 7f8b6a267700 1 -- start start 2023-07-02T08:59:51.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.209+0000 7f8b6a267700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8b64132f50 0x7f8b64133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.209+0000 7f8b6a267700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6409e260 con 0x7f8b6409c580 2023-07-02T08:59:51.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.209+0000 7f8b6a267700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b640a1640 con 0x7f8b64132f50 2023-07-02T08:59:51.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.209+0000 7f8b63fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8b64132f50 0x7f8b64133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.209+0000 7f8b637fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f8b6409c580 0x7f8b6409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55692/0 (socket says 172.21.15.103:55692) 2023-07-02T08:59:51.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.209+0000 7f8b63fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8b64132f50 0x7f8b64133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36952/0 (socket says 172.21.15.103:36952) 2023-07-02T08:59:51.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.209+0000 7f8b637fe700 1 -- 172.21.15.103:0/4163765254 learned_addr learned my addr 172.21.15.103:0/4163765254 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:51.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.210+0000 7f8b63fff700 1 -- 172.21.15.103:0/4163765254 >> v1:172.21.15.103:6789/0 conn(0x7f8b6409c580 legacy=0x7f8b6409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.210+0000 7f8b62ffd700 1 -- 172.21.15.103:0/4163765254 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736117867 0 0) 0x7f8b6409e260 con 0x7f8b6409c580 2023-07-02T08:59:51.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.210+0000 7f8b63fff700 1 -- 172.21.15.103:0/4163765254 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b64132a20 con 0x7f8b64132f50 2023-07-02T08:59:51.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.210+0000 7f8b63fff700 1 --2- 172.21.15.103:0/4163765254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b64132f50 0x7f8b64133360 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3eab0f27a65f2ed server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.210+0000 7f8b62ffd700 1 -- 172.21.15.103:0/4163765254 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8b54009cd0 con 0x7f8b64132f50 2023-07-02T08:59:51.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.210+0000 7f8b62ffd700 1 -- 172.21.15.103:0/4163765254 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8b54009e30 con 0x7f8b64132f50 2023-07-02T08:59:51.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.210+0000 7f8b62ffd700 1 -- 172.21.15.103:0/4163765254 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8b5400e360 con 0x7f8b64132f50 2023-07-02T08:59:51.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.211+0000 7f8b6a267700 1 -- 172.21.15.103:0/4163765254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b64132f50 msgr2=0x7f8b64133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.211+0000 7f8b6a267700 1 --2- 172.21.15.103:0/4163765254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b64132f50 0x7f8b64133360 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.211+0000 7f8b6a267700 1 -- 172.21.15.103:0/4163765254 shutdown_connections 2023-07-02T08:59:51.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.211+0000 7f8b6a267700 1 --2- 172.21.15.103:0/4163765254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b64132f50 0x7f8b64133360 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.211+0000 7f8b6a267700 1 -- 172.21.15.103:0/4163765254 >> 172.21.15.103:0/4163765254 conn(0x7f8b640a2080 msgr2=0x7f8b640a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:51.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.211+0000 7f8b6a267700 1 -- 172.21.15.103:0/4163765254 shutdown_connections 2023-07-02T08:59:51.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.211+0000 7f8b6a267700 1 -- 172.21.15.103:0/4163765254 wait complete. 2023-07-02T08:59:51.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.211+0000 7f8b6a267700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:51.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.213+0000 7f8b6a267700 1 Processor -- start 2023-07-02T08:59:51.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.213+0000 7f8b6a267700 1 -- start start 2023-07-02T08:59:51.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.214+0000 7f8b6a267700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b6409c580 0x7f8b6412beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.214+0000 7f8b6a267700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64133a00 con 0x7f8b6409c580 2023-07-02T08:59:51.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.214+0000 7f8b63fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b6409c580 0x7f8b6412beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.214+0000 7f8b63fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b6409c580 0x7f8b6412beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36964/0 (socket says 172.21.15.103:36964) 2023-07-02T08:59:51.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.214+0000 7f8b63fff700 1 -- 172.21.15.103:0/8411930 learned_addr learned my addr 172.21.15.103:0/8411930 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:51.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.215+0000 7f8b63fff700 1 -- 172.21.15.103:0/8411930 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6406dc00 con 0x7f8b6409c580 2023-07-02T08:59:51.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.215+0000 7f8b63fff700 1 --2- 172.21.15.103:0/8411930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b6409c580 0x7f8b6412beb0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.215+0000 7f8b60ff9700 1 -- 172.21.15.103:0/8411930 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8b54014540 con 0x7f8b6409c580 2023-07-02T08:59:51.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.215+0000 7f8b60ff9700 1 -- 172.21.15.103:0/8411930 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8b540146a0 con 0x7f8b6409c580 2023-07-02T08:59:51.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.215+0000 7f8b60ff9700 1 -- 172.21.15.103:0/8411930 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8b54014940 con 0x7f8b6409c580 2023-07-02T08:59:51.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.215+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6406e920 con 0x7f8b6409c580 2023-07-02T08:59:51.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.215+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b64071f80 con 0x7f8b6409c580 2023-07-02T08:59:51.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.216+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b64061ea0 con 0x7f8b6409c580 2023-07-02T08:59:51.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.217+0000 7f8b60ff9700 1 -- 172.21.15.103:0/8411930 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f8b54014af0 con 0x7f8b6409c580 2023-07-02T08:59:51.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.217+0000 7f8b60ff9700 1 --2- 172.21.15.103:0/8411930 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b500377c0 0x7f8b50039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.217+0000 7f8b60ff9700 1 -- 172.21.15.103:0/8411930 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f8b54019040 con 0x7f8b6409c580 2023-07-02T08:59:51.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.218+0000 7f8b637fe700 1 --2- 172.21.15.103:0/8411930 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b500377c0 0x7f8b50039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.218+0000 7f8b637fe700 1 --2- 172.21.15.103:0/8411930 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b500377c0 0x7f8b50039cd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.227 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.226+0000 7f8b60ff9700 1 -- 172.21.15.103:0/8411930 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f8b54014da0 con 0x7f8b6409c580 2023-07-02T08:59:51.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.242+0000 7f36b5280700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:51.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.247+0000 7f36b5280700 1 Processor -- start 2023-07-02T08:59:51.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.247+0000 7f36b5280700 1 -- start start 2023-07-02T08:59:51.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.248+0000 7f36b5280700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36b0132f50 0x7f36b0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36b5280700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b009e260 con 0x7f36b009c580 2023-07-02T08:59:51.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36b5280700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b00a1640 con 0x7f36b0132f50 2023-07-02T08:59:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36aeffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36b0132f50 0x7f36b0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36ae7fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f36b009c580 0x7f36b009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55698/0 (socket says 172.21.15.103:55698) 2023-07-02T08:59:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36ae7fc700 1 -- 172.21.15.103:0/644333612 learned_addr learned my addr 172.21.15.103:0/644333612 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36aeffd700 1 -- 172.21.15.103:0/644333612 >> v1:172.21.15.103:6789/0 conn(0x7f36b009c580 legacy=0x7f36b009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36adffb700 1 -- 172.21.15.103:0/644333612 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259591622 0 0) 0x7f36b009e260 con 0x7f36b009c580 2023-07-02T08:59:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36aeffd700 1 -- 172.21.15.103:0/644333612 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0132a20 con 0x7f36b0132f50 2023-07-02T08:59:51.252 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.249+0000 7f36aeffd700 1 --2- 172.21.15.103:0/644333612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b0132f50 0x7f36b0133360 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a52b03f0c4d768 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.252 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.250+0000 7f36adffb700 1 -- 172.21.15.103:0/644333612 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36a0009cd0 con 0x7f36b0132f50 2023-07-02T08:59:51.252 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.250+0000 7f36adffb700 1 -- 172.21.15.103:0/644333612 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f36a0009e30 con 0x7f36b0132f50 2023-07-02T08:59:51.252 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.250+0000 7f36adffb700 1 -- 172.21.15.103:0/644333612 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36a000e360 con 0x7f36b0132f50 2023-07-02T08:59:51.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.243+0000 7fc1e0284700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:51.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.251+0000 7f36b5280700 1 -- 172.21.15.103:0/644333612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b0132f50 msgr2=0x7f36b0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.251+0000 7f36b5280700 1 --2- 172.21.15.103:0/644333612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b0132f50 0x7f36b0133360 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.251+0000 7f36b5280700 1 -- 172.21.15.103:0/644333612 shutdown_connections 2023-07-02T08:59:51.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.251+0000 7f36b5280700 1 --2- 172.21.15.103:0/644333612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b0132f50 0x7f36b0133360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.251+0000 7f36b5280700 1 -- 172.21.15.103:0/644333612 >> 172.21.15.103:0/644333612 conn(0x7f36b00a2080 msgr2=0x7f36b00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:51.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.251+0000 7f36b5280700 1 -- 172.21.15.103:0/644333612 shutdown_connections 2023-07-02T08:59:51.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.251+0000 7fc1e0284700 1 Processor -- start 2023-07-02T08:59:51.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.251+0000 7fc1e0284700 1 -- start start 2023-07-02T08:59:51.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7fc1e0284700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc1d8133360 0x7fc1d809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7fc1e0284700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d809e260 con 0x7fc1d8132f50 2023-07-02T08:59:51.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7fc1e0284700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d80a1640 con 0x7fc1d8133360 2023-07-02T08:59:51.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7fc1dd81f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc1d8133360 0x7fc1d809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7f36b5280700 1 -- 172.21.15.103:0/644333612 wait complete. 2023-07-02T08:59:51.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7fc1dd81f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc1d8133360 0x7fc1d809cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36986/0 (socket says 172.21.15.103:36986) 2023-07-02T08:59:51.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7fc1dd81f700 1 -- 172.21.15.103:0/3703710032 learned_addr learned my addr 172.21.15.103:0/3703710032 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:51.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7f36b5280700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:51.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7fc1dd81f700 1 -- 172.21.15.103:0/3703710032 >> v1:172.21.15.103:6789/0 conn(0x7fc1d8132f50 legacy=0x7fc1d809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.252+0000 7fc1dd81f700 1 -- 172.21.15.103:0/3703710032 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d8132a20 con 0x7fc1d8133360 2023-07-02T08:59:51.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.253+0000 7fc1dd81f700 1 --2- 172.21.15.103:0/3703710032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8133360 0x7fc1d809cc90 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34718b49431998d7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.253+0000 7fc1dd01e700 1 -- 172.21.15.103:0/3703710032 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc1d0009d90 con 0x7fc1d8133360 2023-07-02T08:59:51.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.253+0000 7fc1dd01e700 1 -- 172.21.15.103:0/3703710032 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc1d000e230 con 0x7fc1d8133360 2023-07-02T08:59:51.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.253+0000 7fc1dd01e700 1 -- 172.21.15.103:0/3703710032 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc1d000e4d0 con 0x7fc1d8133360 2023-07-02T08:59:51.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.254+0000 7f36b5280700 1 Processor -- start 2023-07-02T08:59:51.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.254+0000 7f36b5280700 1 -- start start 2023-07-02T08:59:51.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.254+0000 7fc1e0284700 1 -- 172.21.15.103:0/3703710032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8133360 msgr2=0x7fc1d809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.254+0000 7fc1e0284700 1 --2- 172.21.15.103:0/3703710032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8133360 0x7fc1d809cc90 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.254+0000 7fc1e0284700 1 -- 172.21.15.103:0/3703710032 shutdown_connections 2023-07-02T08:59:51.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.254+0000 7fc1e0284700 1 --2- 172.21.15.103:0/3703710032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8133360 0x7fc1d809cc90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.254+0000 7fc1e0284700 1 -- 172.21.15.103:0/3703710032 >> 172.21.15.103:0/3703710032 conn(0x7fc1d80a2080 msgr2=0x7fc1d80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:51.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7f36b5280700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b009c580 0x7f36b012beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7f36b5280700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0133a00 con 0x7f36b009c580 2023-07-02T08:59:51.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7f36aeffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b009c580 0x7f36b012beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7f36aeffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b009c580 0x7f36b012beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37000/0 (socket says 172.21.15.103:37000) 2023-07-02T08:59:51.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7f36aeffd700 1 -- 172.21.15.103:0/2115569644 learned_addr learned my addr 172.21.15.103:0/2115569644 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:51.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7fc1e0284700 1 -- 172.21.15.103:0/3703710032 shutdown_connections 2023-07-02T08:59:51.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7fc1e0284700 1 -- 172.21.15.103:0/3703710032 wait complete. 2023-07-02T08:59:51.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7f36aeffd700 1 -- 172.21.15.103:0/2115569644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b006dc00 con 0x7f36b009c580 2023-07-02T08:59:51.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7fc1e0284700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:51.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.255+0000 7f36aeffd700 1 --2- 172.21.15.103:0/2115569644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b009c580 0x7f36b012beb0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.256+0000 7f36977fe700 1 -- 172.21.15.103:0/2115569644 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36a0014540 con 0x7f36b009c580 2023-07-02T08:59:51.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.256+0000 7f36977fe700 1 -- 172.21.15.103:0/2115569644 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f36a00146a0 con 0x7f36b009c580 2023-07-02T08:59:51.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.256+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b006e920 con 0x7f36b009c580 2023-07-02T08:59:51.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.256+0000 7f36977fe700 1 -- 172.21.15.103:0/2115569644 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36a0014940 con 0x7f36b009c580 2023-07-02T08:59:51.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.256+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b0071f80 con 0x7f36b009c580 2023-07-02T08:59:51.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.256+0000 7f36977fe700 1 -- 172.21.15.103:0/2115569644 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f36a0014b40 con 0x7f36b009c580 2023-07-02T08:59:51.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.257+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b0061ea0 con 0x7f36b009c580 2023-07-02T08:59:51.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.257+0000 7fc1e0284700 1 Processor -- start 2023-07-02T08:59:51.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.257+0000 7f36977fe700 1 --2- 172.21.15.103:0/2115569644 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f369c037770 0x7f369c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.257+0000 7f36977fe700 1 -- 172.21.15.103:0/2115569644 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f36a0048390 con 0x7f36b009c580 2023-07-02T08:59:51.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.257+0000 7f36ae7fc700 1 --2- 172.21.15.103:0/2115569644 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f369c037770 0x7f369c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.258+0000 7fc1e0284700 1 -- start start 2023-07-02T08:59:51.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.258+0000 7fc1e0284700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8132f50 0x7fc1d812bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.258+0000 7fc1e0284700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d806d260 con 0x7fc1d8132f50 2023-07-02T08:59:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.258+0000 7f36ae7fc700 1 --2- 172.21.15.103:0/2115569644 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f369c037770 0x7f369c039c80 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.258+0000 7fc1de020700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8132f50 0x7fc1d812bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.258+0000 7fc1de020700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8132f50 0x7fc1d812bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37006/0 (socket says 172.21.15.103:37006) 2023-07-02T08:59:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.258+0000 7fc1de020700 1 -- 172.21.15.103:0/3677043895 learned_addr learned my addr 172.21.15.103:0/3677043895 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.259+0000 7fc1de020700 1 -- 172.21.15.103:0/3677043895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d812bb90 con 0x7fc1d8132f50 2023-07-02T08:59:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.259+0000 7fc1de020700 1 --2- 172.21.15.103:0/3677043895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8132f50 0x7fc1d812bea0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.262+0000 7f36977fe700 1 -- 172.21.15.103:0/2115569644 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f36a000fa40 con 0x7f36b009c580 2023-07-02T08:59:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.262+0000 7fc1ceffd700 1 -- 172.21.15.103:0/3677043895 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc1c800cb80 con 0x7fc1d8132f50 2023-07-02T08:59:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.262+0000 7fc1ceffd700 1 -- 172.21.15.103:0/3677043895 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc1c800cce0 con 0x7fc1d8132f50 2023-07-02T08:59:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.262+0000 7fc1ceffd700 1 -- 172.21.15.103:0/3677043895 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc1c800f230 con 0x7fc1d8132f50 2023-07-02T08:59:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.262+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d806f730 con 0x7fc1d8132f50 2023-07-02T08:59:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.262+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d806f490 con 0x7fc1d8132f50 2023-07-02T08:59:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.263+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d8061ea0 con 0x7fc1d8132f50 2023-07-02T08:59:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.264+0000 7fc1ceffd700 1 -- 172.21.15.103:0/3677043895 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fc1c800f390 con 0x7fc1d8132f50 2023-07-02T08:59:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.264+0000 7fc1ceffd700 1 --2- 172.21.15.103:0/3677043895 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc1c4037800 0x7fc1c4039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.264+0000 7fc1ceffd700 1 -- 172.21.15.103:0/3677043895 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc1c8047720 con 0x7fc1d8132f50 2023-07-02T08:59:51.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.269+0000 7fc1dd81f700 1 --2- 172.21.15.103:0/3677043895 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc1c4037800 0x7fc1c4039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.269+0000 7fc1dd81f700 1 --2- 172.21.15.103:0/3677043895 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc1c4037800 0x7fc1c4039d10 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.273+0000 7fc1ceffd700 1 -- 172.21.15.103:0/3677043895 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc1c800ed50 con 0x7fc1d8132f50 2023-07-02T08:59:51.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.289+0000 7faf8e228700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:51.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.294+0000 7faf8e228700 1 Processor -- start 2023-07-02T08:59:51.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.294+0000 7faf8e228700 1 -- start start 2023-07-02T08:59:51.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf8e228700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf88132f50 0x7faf88133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf8e228700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8809e260 con 0x7faf8809c580 2023-07-02T08:59:51.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf8e228700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf880a1640 con 0x7faf88132f50 2023-07-02T08:59:51.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf877fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf88132f50 0x7faf88133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf877fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf88132f50 0x7faf88133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37018/0 (socket says 172.21.15.103:37018) 2023-07-02T08:59:51.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf877fe700 1 -- 172.21.15.103:0/2092113394 learned_addr learned my addr 172.21.15.103:0/2092113394 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:51.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf86ffd700 1 --1- 172.21.15.103:0/2092113394 >> v1:172.21.15.103:6789/0 conn(0x7faf8809c580 0x7faf8809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55712/0 (socket says 172.21.15.103:55712) 2023-07-02T08:59:51.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf877fe700 1 -- 172.21.15.103:0/2092113394 >> v1:172.21.15.103:6789/0 conn(0x7faf8809c580 legacy=0x7faf8809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf877fe700 1 -- 172.21.15.103:0/2092113394 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf88132a20 con 0x7faf88132f50 2023-07-02T08:59:51.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf877fe700 1 --2- 172.21.15.103:0/2092113394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf88132f50 0x7faf88133360 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0b9909c8abb292 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf867fc700 1 -- 172.21.15.103:0/2092113394 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf78009d90 con 0x7faf88132f50 2023-07-02T08:59:51.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf867fc700 1 -- 172.21.15.103:0/2092113394 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7faf7800e230 con 0x7faf88132f50 2023-07-02T08:59:51.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.295+0000 7faf867fc700 1 -- 172.21.15.103:0/2092113394 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf7800e4d0 con 0x7faf88132f50 2023-07-02T08:59:51.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.296+0000 7faf8e228700 1 -- 172.21.15.103:0/2092113394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf88132f50 msgr2=0x7faf88133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.296+0000 7faf8e228700 1 --2- 172.21.15.103:0/2092113394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf88132f50 0x7faf88133360 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.296+0000 7faf8e228700 1 -- 172.21.15.103:0/2092113394 shutdown_connections 2023-07-02T08:59:51.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.296+0000 7faf8e228700 1 --2- 172.21.15.103:0/2092113394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf88132f50 0x7faf88133360 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.296+0000 7faf8e228700 1 -- 172.21.15.103:0/2092113394 >> 172.21.15.103:0/2092113394 conn(0x7faf880a2080 msgr2=0x7faf880a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:51.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.296+0000 7faf8e228700 1 -- 172.21.15.103:0/2092113394 shutdown_connections 2023-07-02T08:59:51.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.296+0000 7faf8e228700 1 -- 172.21.15.103:0/2092113394 wait complete. 2023-07-02T08:59:51.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.297+0000 7faf8e228700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:51.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.298+0000 7faf8e228700 1 Processor -- start 2023-07-02T08:59:51.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.298+0000 7faf8e228700 1 -- start start 2023-07-02T08:59:51.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.299+0000 7faf8e228700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8809c580 0x7faf8812b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.299+0000 7faf8e228700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8806f9d0 con 0x7faf8809c580 2023-07-02T08:59:51.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.299+0000 7faf877fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8809c580 0x7faf8812b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.299+0000 7faf877fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8809c580 0x7faf8812b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37022/0 (socket says 172.21.15.103:37022) 2023-07-02T08:59:51.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.299+0000 7faf877fe700 1 -- 172.21.15.103:0/448486190 learned_addr learned my addr 172.21.15.103:0/448486190 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:51.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.299+0000 7faf877fe700 1 -- 172.21.15.103:0/448486190 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf88133a00 con 0x7faf8809c580 2023-07-02T08:59:51.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.299+0000 7faf877fe700 1 --2- 172.21.15.103:0/448486190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8809c580 0x7faf8812b7b0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.300+0000 7faf6ffff700 1 -- 172.21.15.103:0/448486190 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf78014540 con 0x7faf8809c580 2023-07-02T08:59:51.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.300+0000 7faf6ffff700 1 -- 172.21.15.103:0/448486190 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7faf780146a0 con 0x7faf8809c580 2023-07-02T08:59:51.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.300+0000 7faf6ffff700 1 -- 172.21.15.103:0/448486190 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf78014940 con 0x7faf8809c580 2023-07-02T08:59:51.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.300+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8806d160 con 0x7faf8809c580 2023-07-02T08:59:51.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.300+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8806da70 con 0x7faf8809c580 2023-07-02T08:59:51.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.301+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf88061ea0 con 0x7faf8809c580 2023-07-02T08:59:51.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.302+0000 7faf6ffff700 1 -- 172.21.15.103:0/448486190 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7faf78014af0 con 0x7faf8809c580 2023-07-02T08:59:51.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.302+0000 7faf6ffff700 1 --2- 172.21.15.103:0/448486190 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf740377b0 0x7faf74039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:51.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.302+0000 7faf86ffd700 1 --2- 172.21.15.103:0/448486190 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf740377b0 0x7faf74039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:51.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.303+0000 7faf86ffd700 1 --2- 172.21.15.103:0/448486190 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf740377b0 0x7faf74039cc0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:51.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.304+0000 7faf6ffff700 1 -- 172.21.15.103:0/448486190 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7faf78019040 con 0x7faf8809c580 2023-07-02T08:59:51.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.306+0000 7faf6ffff700 1 -- 172.21.15.103:0/448486190 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7faf78014da0 con 0x7faf8809c580 2023-07-02T08:59:51.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~6s 2023-07-02T08:59:51.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.466+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8b640725b0 con 0x7f8b6409c580 2023-07-02T08:59:51.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.467+0000 7f8b60ff9700 1 -- 172.21.15.103:0/8411930 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f8b5400fa90 con 0x7f8b6409c580 2023-07-02T08:59:51.468 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:51.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.468+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f36b00725b0 con 0x7f36b009c580 2023-07-02T08:59:51.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b500377c0 msgr2=0x7f8b50039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 --2- 172.21.15.103:0/8411930 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b500377c0 0x7f8b50039cd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f36977fe700 1 -- 172.21.15.103:0/2115569644 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f36a0011320 con 0x7f36b009c580 2023-07-02T08:59:51.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b6409c580 msgr2=0x7f8b6412beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 --2- 172.21.15.103:0/8411930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b6409c580 0x7f8b6412beb0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 shutdown_connections 2023-07-02T08:59:51.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 --2- 172.21.15.103:0/8411930 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b500377c0 0x7f8b50039cd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 --2- 172.21.15.103:0/8411930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b6409c580 0x7f8b6412beb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 >> 172.21.15.103:0/8411930 conn(0x7f8b640a2080 msgr2=0x7f8b641ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:51.472 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:51.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.469+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 shutdown_connections 2023-07-02T08:59:51.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.470+0000 7f8b6a267700 1 -- 172.21.15.103:0/8411930 wait complete. 2023-07-02T08:59:51.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f369c037770 msgr2=0x7f369c039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 --2- 172.21.15.103:0/2115569644 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f369c037770 0x7f369c039c80 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b009c580 msgr2=0x7f36b012beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 --2- 172.21.15.103:0/2115569644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b009c580 0x7f36b012beb0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 shutdown_connections 2023-07-02T08:59:51.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 --2- 172.21.15.103:0/2115569644 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f369c037770 0x7f369c039c80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 --2- 172.21.15.103:0/2115569644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36b009c580 0x7f36b012beb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 >> 172.21.15.103:0/2115569644 conn(0x7f36b00a2080 msgr2=0x7f36b01ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:51.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 shutdown_connections 2023-07-02T08:59:51.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.472+0000 7f36b5280700 1 -- 172.21.15.103:0/2115569644 wait complete. 2023-07-02T08:59:51.492 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:51.495 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:51.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.512+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc1d806f490 con 0x7fc1d8132f50 2023-07-02T08:59:51.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.512+0000 7fc1ceffd700 1 -- 172.21.15.103:0/3677043895 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fc1d806f490 con 0x7fc1d8132f50 2023-07-02T08:59:51.514 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:51.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.514+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc1c4037800 msgr2=0x7fc1c4039d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.514+0000 7fc1e0284700 1 --2- 172.21.15.103:0/3677043895 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc1c4037800 0x7fc1c4039d10 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.514+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8132f50 msgr2=0x7fc1d812bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.514+0000 7fc1e0284700 1 --2- 172.21.15.103:0/3677043895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8132f50 0x7fc1d812bea0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.514+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 shutdown_connections 2023-07-02T08:59:51.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.515+0000 7fc1e0284700 1 --2- 172.21.15.103:0/3677043895 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc1c4037800 0x7fc1c4039d10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.515+0000 7fc1e0284700 1 --2- 172.21.15.103:0/3677043895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8132f50 0x7fc1d812bea0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.515+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 >> 172.21.15.103:0/3677043895 conn(0x7fc1d80a2080 msgr2=0x7fc1d81bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:51.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.515+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 shutdown_connections 2023-07-02T08:59:51.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.515+0000 7fc1e0284700 1 -- 172.21.15.103:0/3677043895 wait complete. 2023-07-02T08:59:51.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.517+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7faf880725b0 con 0x7faf8809c580 2023-07-02T08:59:51.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.517+0000 7faf6ffff700 1 -- 172.21.15.103:0/448486190 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7faf78010e60 con 0x7faf8809c580 2023-07-02T08:59:51.519 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:51.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.519+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf740377b0 msgr2=0x7faf74039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.519+0000 7faf8e228700 1 --2- 172.21.15.103:0/448486190 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf740377b0 0x7faf74039cc0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.519+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8809c580 msgr2=0x7faf8812b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:51.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.519+0000 7faf8e228700 1 --2- 172.21.15.103:0/448486190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8809c580 0x7faf8812b7b0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.519+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 shutdown_connections 2023-07-02T08:59:51.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.519+0000 7faf8e228700 1 --2- 172.21.15.103:0/448486190 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf740377b0 0x7faf74039cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.519+0000 7faf8e228700 1 --2- 172.21.15.103:0/448486190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8809c580 0x7faf8812b7b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:51.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.519+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 >> 172.21.15.103:0/448486190 conn(0x7faf880a2080 msgr2=0x7faf881babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:51.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.520+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 shutdown_connections 2023-07-02T08:59:51.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:51.520+0000 7faf8e228700 1 -- 172.21.15.103:0/448486190 wait complete. 2023-07-02T08:59:51.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:51.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:52.494 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:52.496 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:52.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:52.536 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:52.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.652+0000 7f01b9341700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:52.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.658+0000 7f01b9341700 1 Processor -- start 2023-07-02T08:59:52.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.658+0000 7f01b9341700 1 -- start start 2023-07-02T08:59:52.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.653+0000 7f56994cf700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:52.660 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.659+0000 7f01b9341700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f01b4132f50 0x7f01b4133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.660 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.659+0000 7f01b9341700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b409e260 con 0x7f01b409c580 2023-07-02T08:59:52.660 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.659+0000 7f01b9341700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b40a1640 con 0x7f01b4132f50 2023-07-02T08:59:52.661 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.659+0000 7f56994cf700 1 Processor -- start 2023-07-02T08:59:52.661 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.659+0000 7f56994cf700 1 -- start start 2023-07-02T08:59:52.661 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f56994cf700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f569412f2d0 0x7f569412f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f56994cf700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569412ec50 con 0x7f5694098460 2023-07-02T08:59:52.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f56994cf700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569409a280 con 0x7f569412f2d0 2023-07-02T08:59:52.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.659+0000 7f01b2ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f01b4132f50 0x7f01b4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.659+0000 7f01b2ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f01b4132f50 0x7f01b4133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37032/0 (socket says 172.21.15.103:37032) 2023-07-02T08:59:52.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.659+0000 7f01b2ffd700 1 -- 172.21.15.103:0/3084708002 learned_addr learned my addr 172.21.15.103:0/3084708002 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:52.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f01b2ffd700 1 -- 172.21.15.103:0/3084708002 >> v1:172.21.15.103:6789/0 conn(0x7f01b409c580 legacy=0x7f01b409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f01b2ffd700 1 -- 172.21.15.103:0/3084708002 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b4132a20 con 0x7f01b4132f50 2023-07-02T08:59:52.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f01b2ffd700 1 --2- 172.21.15.103:0/3084708002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b4132f50 0x7f01b4133360 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a28925c72a15960 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f01b1ffb700 1 -- 172.21.15.103:0/3084708002 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f01a4009d90 con 0x7f01b4132f50 2023-07-02T08:59:52.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f01b1ffb700 1 -- 172.21.15.103:0/3084708002 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f01a40159d0 con 0x7f01b4132f50 2023-07-02T08:59:52.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f01b1ffb700 1 -- 172.21.15.103:0/3084708002 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f01a4015c70 con 0x7f01b4132f50 2023-07-02T08:59:52.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f5692ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f569412f2d0 0x7f569412f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f5692ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f569412f2d0 0x7f569412f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37044/0 (socket says 172.21.15.103:37044) 2023-07-02T08:59:52.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.660+0000 7f5692ffd700 1 -- 172.21.15.103:0/4240087582 learned_addr learned my addr 172.21.15.103:0/4240087582 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:52.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f5692ffd700 1 -- 172.21.15.103:0/4240087582 >> v1:172.21.15.103:6789/0 conn(0x7f5694098460 legacy=0x7f5694098870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f5692ffd700 1 -- 172.21.15.103:0/4240087582 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5694099fe0 con 0x7f569412f2d0 2023-07-02T08:59:52.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f5692ffd700 1 --2- 172.21.15.103:0/4240087582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569412f2d0 0x7f569412f6e0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baef12895f9a852f server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f5691ffb700 1 -- 172.21.15.103:0/4240087582 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5684009d90 con 0x7f569412f2d0 2023-07-02T08:59:52.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f5691ffb700 1 -- 172.21.15.103:0/4240087582 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f568400f340 con 0x7f569412f2d0 2023-07-02T08:59:52.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f5691ffb700 1 -- 172.21.15.103:0/4240087582 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f568400f5e0 con 0x7f569412f2d0 2023-07-02T08:59:52.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f01b9341700 1 -- 172.21.15.103:0/3084708002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b4132f50 msgr2=0x7f01b4133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f01b9341700 1 --2- 172.21.15.103:0/3084708002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b4132f50 0x7f01b4133360 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f01b9341700 1 -- 172.21.15.103:0/3084708002 shutdown_connections 2023-07-02T08:59:52.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f01b9341700 1 --2- 172.21.15.103:0/3084708002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b4132f50 0x7f01b4133360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f01b9341700 1 -- 172.21.15.103:0/3084708002 >> 172.21.15.103:0/3084708002 conn(0x7f01b40a2080 msgr2=0x7f01b40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:52.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f01b9341700 1 -- 172.21.15.103:0/3084708002 shutdown_connections 2023-07-02T08:59:52.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.661+0000 7f01b9341700 1 -- 172.21.15.103:0/3084708002 wait complete. 2023-07-02T08:59:52.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.662+0000 7f01b9341700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:52.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.662+0000 7f56994cf700 1 -- 172.21.15.103:0/4240087582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569412f2d0 msgr2=0x7f569412f6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.662+0000 7f56994cf700 1 --2- 172.21.15.103:0/4240087582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569412f2d0 0x7f569412f6e0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.662+0000 7f56994cf700 1 -- 172.21.15.103:0/4240087582 shutdown_connections 2023-07-02T08:59:52.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.662+0000 7f56994cf700 1 --2- 172.21.15.103:0/4240087582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f569412f2d0 0x7f569412f6e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.662+0000 7f56994cf700 1 -- 172.21.15.103:0/4240087582 >> 172.21.15.103:0/4240087582 conn(0x7f56940a1030 msgr2=0x7f56940a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:52.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.662+0000 7f56994cf700 1 -- 172.21.15.103:0/4240087582 shutdown_connections 2023-07-02T08:59:52.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.662+0000 7f56994cf700 1 -- 172.21.15.103:0/4240087582 wait complete. 2023-07-02T08:59:52.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.663+0000 7f56994cf700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:52.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.663+0000 7f01b9341700 1 Processor -- start 2023-07-02T08:59:52.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.663+0000 7f01b9341700 1 -- start start 2023-07-02T08:59:52.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.664+0000 7f01b9341700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b409c580 0x7f01b41c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.664+0000 7f01b9341700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b41c2fc0 con 0x7f01b409c580 2023-07-02T08:59:52.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.664+0000 7f01b2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b409c580 0x7f01b41c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.664+0000 7f01b2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b409c580 0x7f01b41c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37056/0 (socket says 172.21.15.103:37056) 2023-07-02T08:59:52.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.664+0000 7f56994cf700 1 Processor -- start 2023-07-02T08:59:52.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.664+0000 7f01b2ffd700 1 -- 172.21.15.103:0/322836999 learned_addr learned my addr 172.21.15.103:0/322836999 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:52.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.664+0000 7f56994cf700 1 -- start start 2023-07-02T08:59:52.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.664+0000 7f01b2ffd700 1 -- 172.21.15.103:0/322836999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b406d700 con 0x7f01b409c580 2023-07-02T08:59:52.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.665+0000 7f01b2ffd700 1 --2- 172.21.15.103:0/322836999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b409c580 0x7f01b41c3340 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.665+0000 7f019b7fe700 1 -- 172.21.15.103:0/322836999 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f01a401b040 con 0x7f01b409c580 2023-07-02T08:59:52.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.665+0000 7f019b7fe700 1 -- 172.21.15.103:0/322836999 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f01a400caf0 con 0x7f01b409c580 2023-07-02T08:59:52.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.665+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b406ebf0 con 0x7f01b409c580 2023-07-02T08:59:52.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.665+0000 7f56994cf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5694098460 0x7f569406f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.665+0000 7f56994cf700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569408eac0 con 0x7f5694098460 2023-07-02T08:59:52.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.665+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b4132140 con 0x7f01b409c580 2023-07-02T08:59:52.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.665+0000 7f019b7fe700 1 -- 172.21.15.103:0/322836999 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f01a400cd90 con 0x7f01b409c580 2023-07-02T08:59:52.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.666+0000 7f019b7fe700 1 -- 172.21.15.103:0/322836999 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f01a401e040 con 0x7f01b409c580 2023-07-02T08:59:52.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.666+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01940050b0 con 0x7f01b409c580 2023-07-02T08:59:52.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.666+0000 7f019b7fe700 1 --2- 172.21.15.103:0/322836999 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f01a00377c0 0x7f01a0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.666+0000 7f5692ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5694098460 0x7f569406f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.666+0000 7f5692ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5694098460 0x7f569406f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37072/0 (socket says 172.21.15.103:37072) 2023-07-02T08:59:52.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.666+0000 7f5692ffd700 1 -- 172.21.15.103:0/1261313609 learned_addr learned my addr 172.21.15.103:0/1261313609 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:52.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.667+0000 7f5692ffd700 1 -- 172.21.15.103:0/1261313609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5694070690 con 0x7f5694098460 2023-07-02T08:59:52.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.667+0000 7f5692ffd700 1 --2- 172.21.15.103:0/1261313609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5694098460 0x7f569406f650 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.666+0000 7f019b7fe700 1 -- 172.21.15.103:0/322836999 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f01a404fd80 con 0x7f01b409c580 2023-07-02T08:59:52.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.667+0000 7f01b27fc700 1 --2- 172.21.15.103:0/322836999 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f01a00377c0 0x7f01a0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.667+0000 7f01b27fc700 1 --2- 172.21.15.103:0/322836999 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f01a00377c0 0x7f01a0039cd0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.671+0000 7f568b7fe700 1 -- 172.21.15.103:0/1261313609 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f568400fe80 con 0x7f5694098460 2023-07-02T08:59:52.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.671+0000 7f568b7fe700 1 -- 172.21.15.103:0/1261313609 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5684011420 con 0x7f5694098460 2023-07-02T08:59:52.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.671+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569406f140 con 0x7f5694098460 2023-07-02T08:59:52.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.671+0000 7f568b7fe700 1 -- 172.21.15.103:0/1261313609 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f56840116c0 con 0x7f5694098460 2023-07-02T08:59:52.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.671+0000 7f019b7fe700 1 -- 172.21.15.103:0/322836999 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f01a401a9b0 con 0x7f01b409c580 2023-07-02T08:59:52.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.671+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569408e5f0 con 0x7f5694098460 2023-07-02T08:59:52.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.672+0000 7f568b7fe700 1 -- 172.21.15.103:0/1261313609 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f5684011870 con 0x7f5694098460 2023-07-02T08:59:52.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.672+0000 7f568b7fe700 1 --2- 172.21.15.103:0/1261313609 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f56740377b0 0x7f5674039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.673+0000 7f568b7fe700 1 -- 172.21.15.103:0/1261313609 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f5684047910 con 0x7f5694098460 2023-07-02T08:59:52.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.673+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56940710b0 con 0x7f5694098460 2023-07-02T08:59:52.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.673+0000 7f56927fc700 1 --2- 172.21.15.103:0/1261313609 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f56740377b0 0x7f5674039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.673+0000 7f56927fc700 1 --2- 172.21.15.103:0/1261313609 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f56740377b0 0x7f5674039cc0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.678+0000 7f568b7fe700 1 -- 172.21.15.103:0/1261313609 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f5684048e60 con 0x7f5694098460 2023-07-02T08:59:52.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.687+0000 7f36a4084700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:52.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.696+0000 7f36a4084700 1 Processor -- start 2023-07-02T08:59:52.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.697+0000 7f36a4084700 1 -- start start 2023-07-02T08:59:52.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a4084700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f369c133360 0x7f369c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a4084700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c09e260 con 0x7f369c132f50 2023-07-02T08:59:52.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a4084700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369c0a1640 con 0x7f369c133360 2023-07-02T08:59:52.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a161f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f369c133360 0x7f369c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a1e20700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f369c132f50 0x7f369c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55734/0 (socket says 172.21.15.103:55734) 2023-07-02T08:59:52.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a1e20700 1 -- 172.21.15.103:0/4046078286 learned_addr learned my addr 172.21.15.103:0/4046078286 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:52.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a161f700 1 -- 172.21.15.103:0/4046078286 >> v1:172.21.15.103:6789/0 conn(0x7f369c132f50 legacy=0x7f369c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a161f700 1 -- 172.21.15.103:0/4046078286 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c132a20 con 0x7f369c133360 2023-07-02T08:59:52.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a161f700 1 --2- 172.21.15.103:0/4046078286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c133360 0x7f369c09cc90 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431980acf238c2c9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a0e1e700 1 -- 172.21.15.103:0/4046078286 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f368c009cd0 con 0x7f369c133360 2023-07-02T08:59:52.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a0e1e700 1 -- 172.21.15.103:0/4046078286 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f368c009e30 con 0x7f369c133360 2023-07-02T08:59:52.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.698+0000 7f36a0e1e700 1 -- 172.21.15.103:0/4046078286 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f368c00e360 con 0x7f369c133360 2023-07-02T08:59:52.702 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.700+0000 7f36a4084700 1 -- 172.21.15.103:0/4046078286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c133360 msgr2=0x7f369c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.702 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.700+0000 7f36a4084700 1 --2- 172.21.15.103:0/4046078286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c133360 0x7f369c09cc90 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.702 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.700+0000 7f36a4084700 1 -- 172.21.15.103:0/4046078286 shutdown_connections 2023-07-02T08:59:52.702 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.700+0000 7f36a4084700 1 --2- 172.21.15.103:0/4046078286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c133360 0x7f369c09cc90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.702 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.700+0000 7f36a4084700 1 -- 172.21.15.103:0/4046078286 >> 172.21.15.103:0/4046078286 conn(0x7f369c0a2080 msgr2=0x7f369c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:52.702 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.700+0000 7f36a4084700 1 -- 172.21.15.103:0/4046078286 shutdown_connections 2023-07-02T08:59:52.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.700+0000 7f36a4084700 1 -- 172.21.15.103:0/4046078286 wait complete. 2023-07-02T08:59:52.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.700+0000 7f36a4084700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:52.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.701+0000 7f36a4084700 1 Processor -- start 2023-07-02T08:59:52.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.701+0000 7f36a4084700 1 -- start start 2023-07-02T08:59:52.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.697+0000 7f2960d82700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:52.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.702+0000 7f36a4084700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c132f50 0x7f369c12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.702+0000 7f36a4084700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c06d260 con 0x7f369c132f50 2023-07-02T08:59:52.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.702+0000 7f36a1e20700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c132f50 0x7f369c12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.702+0000 7f36a1e20700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c132f50 0x7f369c12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37094/0 (socket says 172.21.15.103:37094) 2023-07-02T08:59:52.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.702+0000 7f36a1e20700 1 -- 172.21.15.103:0/1926156487 learned_addr learned my addr 172.21.15.103:0/1926156487 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:52.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.703+0000 7f2960d82700 1 Processor -- start 2023-07-02T08:59:52.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.703+0000 7f2960d82700 1 -- start start 2023-07-02T08:59:52.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.703+0000 7f36a1e20700 1 -- 172.21.15.103:0/1926156487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c12bb90 con 0x7f369c132f50 2023-07-02T08:59:52.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.703+0000 7f36a1e20700 1 --2- 172.21.15.103:0/1926156487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c132f50 0x7f369c12bea0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.703+0000 7f36927fc700 1 -- 172.21.15.103:0/1926156487 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3698005a60 con 0x7f369c132f50 2023-07-02T08:59:52.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.703+0000 7f36927fc700 1 -- 172.21.15.103:0/1926156487 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3698005bc0 con 0x7f369c132f50 2023-07-02T08:59:52.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.703+0000 7f36927fc700 1 -- 172.21.15.103:0/1926156487 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3698005e60 con 0x7f369c132f50 2023-07-02T08:59:52.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.704+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c06f730 con 0x7f369c132f50 2023-07-02T08:59:52.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.704+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c06f490 con 0x7f369c132f50 2023-07-02T08:59:52.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.704+0000 7f2960d82700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f295c133360 0x7f295c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.708 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.704+0000 7f2960d82700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295c09e260 con 0x7f295c132f50 2023-07-02T08:59:52.708 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.704+0000 7f2960d82700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295c0a1640 con 0x7f295c133360 2023-07-02T08:59:52.708 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.705+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c061ea0 con 0x7f369c132f50 2023-07-02T08:59:52.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.706+0000 7f36927fc700 1 -- 172.21.15.103:0/1926156487 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f36980070b0 con 0x7f369c132f50 2023-07-02T08:59:52.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.706+0000 7f36927fc700 1 --2- 172.21.15.103:0/1926156487 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36800377c0 0x7f3680039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.706+0000 7f36927fc700 1 -- 172.21.15.103:0/1926156487 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3698046bb0 con 0x7f369c132f50 2023-07-02T08:59:52.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.706+0000 7f36a161f700 1 --2- 172.21.15.103:0/1926156487 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36800377c0 0x7f3680039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.707+0000 7f36a161f700 1 --2- 172.21.15.103:0/1926156487 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36800377c0 0x7f3680039cd0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.708+0000 7f2959d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f295c133360 0x7f295c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.708+0000 7f2959d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f295c133360 0x7f295c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37098/0 (socket says 172.21.15.103:37098) 2023-07-02T08:59:52.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.708+0000 7f2959d9b700 1 -- 172.21.15.103:0/1001590372 learned_addr learned my addr 172.21.15.103:0/1001590372 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:52.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.708+0000 7f295a59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f295c132f50 0x7f295c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55740/0 (socket says 172.21.15.103:55740) 2023-07-02T08:59:52.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.708+0000 7f2959d9b700 1 -- 172.21.15.103:0/1001590372 >> v1:172.21.15.103:6789/0 conn(0x7f295c132f50 legacy=0x7f295c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.708+0000 7f2959d9b700 1 -- 172.21.15.103:0/1001590372 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c132a20 con 0x7f295c133360 2023-07-02T08:59:52.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.708+0000 7f2959d9b700 1 --2- 172.21.15.103:0/1001590372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c133360 0x7f295c09cc90 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ffdb558b34db9a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.710+0000 7f36927fc700 1 -- 172.21.15.103:0/1926156487 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f369800d710 con 0x7f369c132f50 2023-07-02T08:59:52.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.710+0000 7f295959a700 1 -- 172.21.15.103:0/1001590372 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2950009d90 con 0x7f295c133360 2023-07-02T08:59:52.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.710+0000 7f295959a700 1 -- 172.21.15.103:0/1001590372 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f295000e230 con 0x7f295c133360 2023-07-02T08:59:52.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.710+0000 7f295959a700 1 -- 172.21.15.103:0/1001590372 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f295000e4d0 con 0x7f295c133360 2023-07-02T08:59:52.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.711+0000 7f2960d82700 1 -- 172.21.15.103:0/1001590372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c133360 msgr2=0x7f295c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.711+0000 7f2960d82700 1 --2- 172.21.15.103:0/1001590372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c133360 0x7f295c09cc90 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.711+0000 7f2960d82700 1 -- 172.21.15.103:0/1001590372 shutdown_connections 2023-07-02T08:59:52.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.711+0000 7f2960d82700 1 --2- 172.21.15.103:0/1001590372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c133360 0x7f295c09cc90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.711+0000 7f2960d82700 1 -- 172.21.15.103:0/1001590372 >> 172.21.15.103:0/1001590372 conn(0x7f295c0a2080 msgr2=0x7f295c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:52.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.711+0000 7f2960d82700 1 -- 172.21.15.103:0/1001590372 shutdown_connections 2023-07-02T08:59:52.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.711+0000 7f2960d82700 1 -- 172.21.15.103:0/1001590372 wait complete. 2023-07-02T08:59:52.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.712+0000 7f2960d82700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:52.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.713+0000 7f2960d82700 1 Processor -- start 2023-07-02T08:59:52.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.713+0000 7f2960d82700 1 -- start start 2023-07-02T08:59:52.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.713+0000 7f2960d82700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c132f50 0x7f295c12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.713+0000 7f2960d82700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c06d260 con 0x7f295c132f50 2023-07-02T08:59:52.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.713+0000 7f295a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c132f50 0x7f295c12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.714+0000 7f295a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c132f50 0x7f295c12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37114/0 (socket says 172.21.15.103:37114) 2023-07-02T08:59:52.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.714+0000 7f295a59c700 1 -- 172.21.15.103:0/2547881904 learned_addr learned my addr 172.21.15.103:0/2547881904 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:52.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.714+0000 7f295a59c700 1 -- 172.21.15.103:0/2547881904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c12bb90 con 0x7f295c132f50 2023-07-02T08:59:52.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.714+0000 7f295a59c700 1 --2- 172.21.15.103:0/2547881904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c132f50 0x7f295c12bea0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.714+0000 7f294affd700 1 -- 172.21.15.103:0/2547881904 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f294c0059a0 con 0x7f295c132f50 2023-07-02T08:59:52.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.714+0000 7f294affd700 1 -- 172.21.15.103:0/2547881904 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f294c005b00 con 0x7f295c132f50 2023-07-02T08:59:52.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.714+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c06f730 con 0x7f295c132f50 2023-07-02T08:59:52.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.714+0000 7f294affd700 1 -- 172.21.15.103:0/2547881904 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f294c005da0 con 0x7f295c132f50 2023-07-02T08:59:52.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.715+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c06f490 con 0x7f295c132f50 2023-07-02T08:59:52.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.716+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295c061ea0 con 0x7f295c132f50 2023-07-02T08:59:52.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.717+0000 7f294affd700 1 -- 172.21.15.103:0/2547881904 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f294c0070b0 con 0x7f295c132f50 2023-07-02T08:59:52.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.717+0000 7f294affd700 1 --2- 172.21.15.103:0/2547881904 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f293c0377c0 0x7f293c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:52.719 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.717+0000 7f294affd700 1 -- 172.21.15.103:0/2547881904 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f294c046c10 con 0x7f295c132f50 2023-07-02T08:59:52.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.720+0000 7f2959d9b700 1 --2- 172.21.15.103:0/2547881904 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f293c0377c0 0x7f293c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:52.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.720+0000 7f2959d9b700 1 --2- 172.21.15.103:0/2547881904 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f293c0377c0 0x7f293c039cd0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:52.722 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.721+0000 7f294affd700 1 -- 172.21.15.103:0/2547881904 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f294c00d330 con 0x7f295c132f50 2023-07-02T08:59:52.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.886+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5694072070 con 0x7f5694098460 2023-07-02T08:59:52.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.886+0000 7f568b7fe700 1 -- 172.21.15.103:0/1261313609 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f568400b550 con 0x7f5694098460 2023-07-02T08:59:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:52.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.888+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f56740377b0 msgr2=0x7f5674039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.888+0000 7f56994cf700 1 --2- 172.21.15.103:0/1261313609 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f56740377b0 0x7f5674039cc0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.888+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5694098460 msgr2=0x7f569406f650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.888+0000 7f56994cf700 1 --2- 172.21.15.103:0/1261313609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5694098460 0x7f569406f650 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.889+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 shutdown_connections 2023-07-02T08:59:52.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.889+0000 7f56994cf700 1 --2- 172.21.15.103:0/1261313609 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f56740377b0 0x7f5674039cc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.889+0000 7f56994cf700 1 --2- 172.21.15.103:0/1261313609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5694098460 0x7f569406f650 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.889+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 >> 172.21.15.103:0/1261313609 conn(0x7f56940a1030 msgr2=0x7f569412b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:52.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.889+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 shutdown_connections 2023-07-02T08:59:52.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.889+0000 7f56994cf700 1 -- 172.21.15.103:0/1261313609 wait complete. 2023-07-02T08:59:52.896 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.895+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0194005360 con 0x7f01b409c580 2023-07-02T08:59:52.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.896+0000 7f019b7fe700 1 -- 172.21.15.103:0/322836999 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f01a4007e70 con 0x7f01b409c580 2023-07-02T08:59:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:52.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f01a00377c0 msgr2=0x7f01a0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 --2- 172.21.15.103:0/322836999 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f01a00377c0 0x7f01a0039cd0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b409c580 msgr2=0x7f01b41c3340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 --2- 172.21.15.103:0/322836999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b409c580 0x7f01b41c3340 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 shutdown_connections 2023-07-02T08:59:52.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 --2- 172.21.15.103:0/322836999 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f01a00377c0 0x7f01a0039cd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 --2- 172.21.15.103:0/322836999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01b409c580 0x7f01b41c3340 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 >> 172.21.15.103:0/322836999 conn(0x7f01b40a2080 msgr2=0x7f01b4098280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:52.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 shutdown_connections 2023-07-02T08:59:52.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.898+0000 7f01b9341700 1 -- 172.21.15.103:0/322836999 wait complete. 2023-07-02T08:59:52.905 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2023-07-02T08:59:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.911+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f369c06f490 con 0x7f369c132f50 2023-07-02T08:59:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.911+0000 7f36927fc700 1 -- 172.21.15.103:0/1926156487 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f369c06f490 con 0x7f369c132f50 2023-07-02T08:59:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:52.913 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2023-07-02T08:59:52.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36800377c0 msgr2=0x7f3680039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 --2- 172.21.15.103:0/1926156487 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36800377c0 0x7f3680039cd0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c132f50 msgr2=0x7f369c12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 --2- 172.21.15.103:0/1926156487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c132f50 0x7f369c12bea0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 shutdown_connections 2023-07-02T08:59:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 --2- 172.21.15.103:0/1926156487 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36800377c0 0x7f3680039cd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 --2- 172.21.15.103:0/1926156487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c132f50 0x7f369c12bea0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 >> 172.21.15.103:0/1926156487 conn(0x7f369c0a2080 msgr2=0x7f369c1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:52.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 shutdown_connections 2023-07-02T08:59:52.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.913+0000 7f36a4084700 1 -- 172.21.15.103:0/1926156487 wait complete. 2023-07-02T08:59:52.927 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:52.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.932+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f295c06f490 con 0x7f295c132f50 2023-07-02T08:59:52.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.932+0000 7f294affd700 1 -- 172.21.15.103:0/2547881904 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f295c06f490 con 0x7f295c132f50 2023-07-02T08:59:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:52.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f293c0377c0 msgr2=0x7f293c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 --2- 172.21.15.103:0/2547881904 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f293c0377c0 0x7f293c039cd0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c132f50 msgr2=0x7f295c12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:52.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 --2- 172.21.15.103:0/2547881904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c132f50 0x7f295c12bea0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 shutdown_connections 2023-07-02T08:59:52.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 --2- 172.21.15.103:0/2547881904 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f293c0377c0 0x7f293c039cd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 --2- 172.21.15.103:0/2547881904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f295c132f50 0x7f295c12bea0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:52.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 >> 172.21.15.103:0/2547881904 conn(0x7f295c0a2080 msgr2=0x7f295c1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:52.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.934+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 shutdown_connections 2023-07-02T08:59:52.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:52.935+0000 7f2960d82700 1 -- 172.21.15.103:0/2547881904 wait complete. 2023-07-02T08:59:52.949 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2023-07-02T08:59:53.906 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T08:59:53.914 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T08:59:53.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:53.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T08:59:54.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.055+0000 7fa59aa09700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:54.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.063+0000 7fa59aa09700 1 Processor -- start 2023-07-02T08:59:54.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.063+0000 7fa59aa09700 1 -- start start 2023-07-02T08:59:54.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.064+0000 7fa59aa09700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa59409e3c0 0x7fa5941358a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.064+0000 7fa59aa09700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59409b590 con 0x7fa59409d8a0 2023-07-02T08:59:54.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.064+0000 7fa59aa09700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59409a5b0 con 0x7fa59409e3c0 2023-07-02T08:59:54.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa5937fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa59409e3c0 0x7fa5941358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa593fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa59409d8a0 0x7fa59409dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36976/0 (socket says 172.21.15.103:36976) 2023-07-02T08:59:54.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa5937fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa59409e3c0 0x7fa5941358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43674/0 (socket says 172.21.15.103:43674) 2023-07-02T08:59:54.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa593fff700 1 -- 172.21.15.103:0/1908061690 learned_addr learned my addr 172.21.15.103:0/1908061690 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:54.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa5937fe700 1 -- 172.21.15.103:0/1908061690 >> v1:172.21.15.103:6789/0 conn(0x7fa59409d8a0 legacy=0x7fa59409dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa5937fe700 1 -- 172.21.15.103:0/1908061690 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59409f700 con 0x7fa59409e3c0 2023-07-02T08:59:54.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa5937fe700 1 --2- 172.21.15.103:0/1908061690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409e3c0 0x7fa5941358a0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c5e03002ec174e server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa592ffd700 1 -- 172.21.15.103:0/1908061690 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa588009cd0 con 0x7fa59409e3c0 2023-07-02T08:59:54.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa592ffd700 1 -- 172.21.15.103:0/1908061690 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa588009e30 con 0x7fa59409e3c0 2023-07-02T08:59:54.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.065+0000 7fa592ffd700 1 -- 172.21.15.103:0/1908061690 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa58800e360 con 0x7fa59409e3c0 2023-07-02T08:59:54.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.066+0000 7fa59aa09700 1 -- 172.21.15.103:0/1908061690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409e3c0 msgr2=0x7fa5941358a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.066+0000 7fa59aa09700 1 --2- 172.21.15.103:0/1908061690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409e3c0 0x7fa5941358a0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.067+0000 7fa59aa09700 1 -- 172.21.15.103:0/1908061690 shutdown_connections 2023-07-02T08:59:54.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.067+0000 7fa59aa09700 1 --2- 172.21.15.103:0/1908061690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409e3c0 0x7fa5941358a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.067+0000 7fa59aa09700 1 -- 172.21.15.103:0/1908061690 >> 172.21.15.103:0/1908061690 conn(0x7fa5940a3280 msgr2=0x7fa5940a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:54.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.067+0000 7fa59aa09700 1 -- 172.21.15.103:0/1908061690 shutdown_connections 2023-07-02T08:59:54.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.067+0000 7fa59aa09700 1 -- 172.21.15.103:0/1908061690 wait complete. 2023-07-02T08:59:54.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.067+0000 7fa59aa09700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:54.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.068+0000 7fa59aa09700 1 Processor -- start 2023-07-02T08:59:54.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.069+0000 7fa59aa09700 1 -- start start 2023-07-02T08:59:54.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.070+0000 7fa59aa09700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409d8a0 0x7fa5941c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.070+0000 7fa59aa09700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59411e910 con 0x7fa59409d8a0 2023-07-02T08:59:54.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.070+0000 7fa593fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409d8a0 0x7fa5941c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.070+0000 7fa593fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409d8a0 0x7fa5941c73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43686/0 (socket says 172.21.15.103:43686) 2023-07-02T08:59:54.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.070+0000 7fa593fff700 1 -- 172.21.15.103:0/2759182098 learned_addr learned my addr 172.21.15.103:0/2759182098 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:54.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.070+0000 7fa593fff700 1 -- 172.21.15.103:0/2759182098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59411b830 con 0x7fa59409d8a0 2023-07-02T08:59:54.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.070+0000 7fa593fff700 1 --2- 172.21.15.103:0/2759182098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409d8a0 0x7fa5941c73b0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.070+0000 7fa590ff9700 1 -- 172.21.15.103:0/2759182098 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa584005a60 con 0x7fa59409d8a0 2023-07-02T08:59:54.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.071+0000 7fa590ff9700 1 -- 172.21.15.103:0/2759182098 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa584005bc0 con 0x7fa59409d8a0 2023-07-02T08:59:54.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.071+0000 7fa590ff9700 1 -- 172.21.15.103:0/2759182098 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa584005e60 con 0x7fa59409d8a0 2023-07-02T08:59:54.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.071+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59411c300 con 0x7fa59409d8a0 2023-07-02T08:59:54.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.071+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59411ddc0 con 0x7fa59409d8a0 2023-07-02T08:59:54.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.072+0000 7fa590ff9700 1 -- 172.21.15.103:0/2759182098 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa5840070b0 con 0x7fa59409d8a0 2023-07-02T08:59:54.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.072+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59409b250 con 0x7fa59409d8a0 2023-07-02T08:59:54.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.072+0000 7fa590ff9700 1 --2- 172.21.15.103:0/2759182098 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5800377c0 0x7fa580039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.072+0000 7fa590ff9700 1 -- 172.21.15.103:0/2759182098 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa584046be0 con 0x7fa59409d8a0 2023-07-02T08:59:54.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.072+0000 7fa5937fe700 1 --2- 172.21.15.103:0/2759182098 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5800377c0 0x7fa580039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.073+0000 7fa5937fe700 1 --2- 172.21.15.103:0/2759182098 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5800377c0 0x7fa580039cd0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.077+0000 7fa590ff9700 1 -- 172.21.15.103:0/2759182098 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa58400a4e0 con 0x7fa59409d8a0 2023-07-02T08:59:54.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.096+0000 7f23f7a13700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:54.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.101+0000 7f23f7a13700 1 Processor -- start 2023-07-02T08:59:54.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.102+0000 7f23f7a13700 1 -- start start 2023-07-02T08:59:54.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.102+0000 7f23f7a13700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f23f0133360 0x7f23f009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.102+0000 7f23f7a13700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f009e260 con 0x7f23f0132f50 2023-07-02T08:59:54.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.102+0000 7f23f7a13700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f00a1640 con 0x7f23f0133360 2023-07-02T08:59:54.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.095+0000 7f935c719700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:54.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.102+0000 7f23f57af700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f23f0132f50 0x7f23f009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36978/0 (socket says 172.21.15.103:36978) 2023-07-02T08:59:54.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23f57af700 1 -- 172.21.15.103:0/2023066465 learned_addr learned my addr 172.21.15.103:0/2023066465 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:54.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23f4fae700 1 --2- 172.21.15.103:0/2023066465 >> v2:172.21.15.103:3300/0 conn(0x7f23f0133360 0x7f23f009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23e7fff700 1 -- 172.21.15.103:0/2023066465 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766379810 0 0) 0x7f23f009e260 con 0x7f23f0132f50 2023-07-02T08:59:54.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23e7fff700 1 -- 172.21.15.103:0/2023066465 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23dc003580 con 0x7f23f0132f50 2023-07-02T08:59:54.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23f4fae700 1 -- 172.21.15.103:0/2023066465 >> v1:172.21.15.103:6789/0 conn(0x7f23f0132f50 legacy=0x7f23f009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23f4fae700 1 -- 172.21.15.103:0/2023066465 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f0132a20 con 0x7f23f0133360 2023-07-02T08:59:54.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23f4fae700 1 --2- 172.21.15.103:0/2023066465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0133360 0x7f23f009cc90 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580617178f95142e server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23e7fff700 1 -- 172.21.15.103:0/2023066465 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f23e0009c80 con 0x7f23f0133360 2023-07-02T08:59:54.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23e7fff700 1 -- 172.21.15.103:0/2023066465 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f23e0009de0 con 0x7f23f0133360 2023-07-02T08:59:54.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.103+0000 7f23e7fff700 1 -- 172.21.15.103:0/2023066465 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f23e000e360 con 0x7f23f0133360 2023-07-02T08:59:54.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.104+0000 7f935c719700 1 Processor -- start 2023-07-02T08:59:54.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.104+0000 7f935c719700 1 -- start start 2023-07-02T08:59:54.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.104+0000 7f23f7a13700 1 -- 172.21.15.103:0/2023066465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0133360 msgr2=0x7f23f009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.104+0000 7f23f7a13700 1 --2- 172.21.15.103:0/2023066465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0133360 0x7f23f009cc90 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.104+0000 7f23f7a13700 1 -- 172.21.15.103:0/2023066465 shutdown_connections 2023-07-02T08:59:54.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.104+0000 7f23f7a13700 1 --2- 172.21.15.103:0/2023066465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0133360 0x7f23f009cc90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.104+0000 7f23f7a13700 1 -- 172.21.15.103:0/2023066465 >> 172.21.15.103:0/2023066465 conn(0x7f23f00a2080 msgr2=0x7f23f00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:54.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f23f7a13700 1 -- 172.21.15.103:0/2023066465 shutdown_connections 2023-07-02T08:59:54.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f23f7a13700 1 -- 172.21.15.103:0/2023066465 wait complete. 2023-07-02T08:59:54.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f935c719700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9354133990 0x7f9354135db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f935c719700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9354097380 con 0x7f9354130b20 2023-07-02T08:59:54.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f935c719700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93540a1410 con 0x7f9354133990 2023-07-02T08:59:54.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f23f7a13700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:54.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f9359cb4700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9354133990 0x7f9354135db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f9359cb4700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9354133990 0x7f9354135db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43704/0 (socket says 172.21.15.103:43704) 2023-07-02T08:59:54.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f935a4b5700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9354130b20 0x7f9354130f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36994/0 (socket says 172.21.15.103:36994) 2023-07-02T08:59:54.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f9359cb4700 1 -- 172.21.15.103:0/2530893516 learned_addr learned my addr 172.21.15.103:0/2530893516 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:54.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.106+0000 7f9359cb4700 1 -- 172.21.15.103:0/2530893516 >> v1:172.21.15.103:6789/0 conn(0x7f9354130b20 legacy=0x7f9354130f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.106+0000 7f9359cb4700 1 -- 172.21.15.103:0/2530893516 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93540975b0 con 0x7f9354133990 2023-07-02T08:59:54.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.106+0000 7f93594b3700 1 -- 172.21.15.103:0/2530893516 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976351298 0 0) 0x7f9354097380 con 0x7f9354130b20 2023-07-02T08:59:54.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.106+0000 7f9359cb4700 1 --2- 172.21.15.103:0/2530893516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354133990 0x7f9354135db0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebc89fe9d1e245d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.106+0000 7f23f7a13700 1 Processor -- start 2023-07-02T08:59:54.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.106+0000 7f23f7a13700 1 -- start start 2023-07-02T08:59:54.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.106+0000 7f93594b3700 1 -- 172.21.15.103:0/2530893516 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9344009d90 con 0x7f9354133990 2023-07-02T08:59:54.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f93594b3700 1 -- 172.21.15.103:0/2530893516 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f934400e230 con 0x7f9354133990 2023-07-02T08:59:54.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f93594b3700 1 -- 172.21.15.103:0/2530893516 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f934400e4d0 con 0x7f9354133990 2023-07-02T08:59:54.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f23f7a13700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0132f50 0x7f23f012c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f23f7a13700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f012beb0 con 0x7f23f0132f50 2023-07-02T08:59:54.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f23f57af700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0132f50 0x7f23f012c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f23f57af700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0132f50 0x7f23f012c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43716/0 (socket says 172.21.15.103:43716) 2023-07-02T08:59:54.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f23f57af700 1 -- 172.21.15.103:0/4191621723 learned_addr learned my addr 172.21.15.103:0/4191621723 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:54.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f23f57af700 1 -- 172.21.15.103:0/4191621723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f006fb30 con 0x7f23f0132f50 2023-07-02T08:59:54.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.107+0000 7f23f57af700 1 --2- 172.21.15.103:0/4191621723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0132f50 0x7f23f012c270 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f23e5ffb700 1 -- 172.21.15.103:0/4191621723 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f23ec005a60 con 0x7f23f0132f50 2023-07-02T08:59:54.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f23e5ffb700 1 -- 172.21.15.103:0/4191621723 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f23ec005bc0 con 0x7f23f0132f50 2023-07-02T08:59:54.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f935c719700 1 -- 172.21.15.103:0/2530893516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354133990 msgr2=0x7f9354135db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f935c719700 1 --2- 172.21.15.103:0/2530893516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354133990 0x7f9354135db0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f935c719700 1 -- 172.21.15.103:0/2530893516 shutdown_connections 2023-07-02T08:59:54.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f935c719700 1 --2- 172.21.15.103:0/2530893516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354133990 0x7f9354135db0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f935c719700 1 -- 172.21.15.103:0/2530893516 >> 172.21.15.103:0/2530893516 conn(0x7f935409f500 msgr2=0x7f935409f910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:54.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f935c719700 1 -- 172.21.15.103:0/2530893516 shutdown_connections 2023-07-02T08:59:54.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f23e5ffb700 1 -- 172.21.15.103:0/4191621723 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f23ec005e60 con 0x7f23f0132f50 2023-07-02T08:59:54.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f012bb10 con 0x7f23f0132f50 2023-07-02T08:59:54.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f006f5f0 con 0x7f23f0132f50 2023-07-02T08:59:54.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.109+0000 7f23e5ffb700 1 -- 172.21.15.103:0/4191621723 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f23ec0070b0 con 0x7f23f0132f50 2023-07-02T08:59:54.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.109+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f012bd00 con 0x7f23f0132f50 2023-07-02T08:59:54.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.109+0000 7f23e5ffb700 1 --2- 172.21.15.103:0/4191621723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23d40377c0 0x7f23d4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.109+0000 7f23e5ffb700 1 -- 172.21.15.103:0/4191621723 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f23ec046ba0 con 0x7f23f0132f50 2023-07-02T08:59:54.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f935c719700 1 -- 172.21.15.103:0/2530893516 wait complete. 2023-07-02T08:59:54.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.108+0000 7f935c719700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:54.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.110+0000 7f935c719700 1 Processor -- start 2023-07-02T08:59:54.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.110+0000 7f935c719700 1 -- start start 2023-07-02T08:59:54.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.111+0000 7f935c719700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354130b20 0x7f93541d3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.111+0000 7f935c719700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93541bd970 con 0x7f9354130b20 2023-07-02T08:59:54.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.112+0000 7f23f4fae700 1 --2- 172.21.15.103:0/4191621723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23d40377c0 0x7f23d4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.105+0000 7f75c5bee700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:54.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.112+0000 7f23f4fae700 1 --2- 172.21.15.103:0/4191621723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23d40377c0 0x7f23d4039cd0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.113+0000 7f75c5bee700 1 Processor -- start 2023-07-02T08:59:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.113+0000 7f75c5bee700 1 -- start start 2023-07-02T08:59:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75c5bee700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f75c009d8a0 0x7f75c009dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75c5bee700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c009b590 con 0x7f75c009e1f0 2023-07-02T08:59:54.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75c5bee700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c009a5b0 con 0x7f75c009d8a0 2023-07-02T08:59:54.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75bf7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f75c009d8a0 0x7f75c009dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75beffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f75c009e1f0 0x7f75c0130660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37008/0 (socket says 172.21.15.103:37008) 2023-07-02T08:59:54.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75beffd700 1 -- 172.21.15.103:0/1027983406 learned_addr learned my addr 172.21.15.103:0/1027983406 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:54.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75bf7fe700 1 -- 172.21.15.103:0/1027983406 >> v1:172.21.15.103:6789/0 conn(0x7f75c009e1f0 legacy=0x7f75c0130660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75bf7fe700 1 -- 172.21.15.103:0/1027983406 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c009f700 con 0x7f75c009d8a0 2023-07-02T08:59:54.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.114+0000 7f75bf7fe700 1 --2- 172.21.15.103:0/1027983406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c009dcb0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9d587513c21faf server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f935a4b5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354130b20 0x7f93541d3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f935a4b5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354130b20 0x7f93541d3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43732/0 (socket says 172.21.15.103:43732) 2023-07-02T08:59:54.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f935a4b5700 1 -- 172.21.15.103:0/712287083 learned_addr learned my addr 172.21.15.103:0/712287083 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:54.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f75be7fc700 1 -- 172.21.15.103:0/1027983406 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f75b0009cd0 con 0x7f75c009d8a0 2023-07-02T08:59:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f75be7fc700 1 -- 172.21.15.103:0/1027983406 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f75b0009e30 con 0x7f75c009d8a0 2023-07-02T08:59:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f75be7fc700 1 -- 172.21.15.103:0/1027983406 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f75b000e360 con 0x7f75c009d8a0 2023-07-02T08:59:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f23e5ffb700 1 -- 172.21.15.103:0/4191621723 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f23ec00a300 con 0x7f23f0132f50 2023-07-02T08:59:54.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f935a4b5700 1 -- 172.21.15.103:0/712287083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9354066dd0 con 0x7f9354130b20 2023-07-02T08:59:54.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f935a4b5700 1 --2- 172.21.15.103:0/712287083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354130b20 0x7f93541d3d70 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f934affd700 1 -- 172.21.15.103:0/712287083 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9350004cf0 con 0x7f9354130b20 2023-07-02T08:59:54.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f934affd700 1 -- 172.21.15.103:0/712287083 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9350004e50 con 0x7f9354130b20 2023-07-02T08:59:54.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f934affd700 1 -- 172.21.15.103:0/712287083 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f935000d280 con 0x7f9354130b20 2023-07-02T08:59:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.116+0000 7f75c5bee700 1 -- 172.21.15.103:0/1027983406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 msgr2=0x7f75c009dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.116+0000 7f75c5bee700 1 --2- 172.21.15.103:0/1027983406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c009dcb0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.116+0000 7f75c5bee700 1 -- 172.21.15.103:0/1027983406 shutdown_connections 2023-07-02T08:59:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.116+0000 7f75c5bee700 1 --2- 172.21.15.103:0/1027983406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c009dcb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.116+0000 7f75c5bee700 1 -- 172.21.15.103:0/1027983406 >> 172.21.15.103:0/1027983406 conn(0x7f75c00a3280 msgr2=0x7f75c00a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:54.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.115+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93541bd660 con 0x7f9354130b20 2023-07-02T08:59:54.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.116+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935412e960 con 0x7f9354130b20 2023-07-02T08:59:54.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.117+0000 7f75c5bee700 1 -- 172.21.15.103:0/1027983406 shutdown_connections 2023-07-02T08:59:54.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.117+0000 7f75c5bee700 1 -- 172.21.15.103:0/1027983406 wait complete. 2023-07-02T08:59:54.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.117+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935412ec10 con 0x7f9354130b20 2023-07-02T08:59:54.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.117+0000 7f75c5bee700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:54.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.117+0000 7f934affd700 1 -- 172.21.15.103:0/712287083 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f935000d430 con 0x7f9354130b20 2023-07-02T08:59:54.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.117+0000 7f934affd700 1 --2- 172.21.15.103:0/712287083 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f93380377c0 0x7f9338039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.117+0000 7f9359cb4700 1 --2- 172.21.15.103:0/712287083 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f93380377c0 0x7f9338039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.118+0000 7f934affd700 1 -- 172.21.15.103:0/712287083 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9350005870 con 0x7f9354130b20 2023-07-02T08:59:54.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.118+0000 7f9359cb4700 1 --2- 172.21.15.103:0/712287083 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f93380377c0 0x7f9338039cd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.118+0000 7f75c5bee700 1 Processor -- start 2023-07-02T08:59:54.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.118+0000 7f75c5bee700 1 -- start start 2023-07-02T08:59:54.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.119+0000 7f75c5bee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c01c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.119+0000 7f75c5bee700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c006e570 con 0x7f75c009d8a0 2023-07-02T08:59:54.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.120+0000 7f75bf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c01c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.120+0000 7f75bf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c01c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43734/0 (socket says 172.21.15.103:43734) 2023-07-02T08:59:54.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.120+0000 7f75bf7fe700 1 -- 172.21.15.103:0/4083074569 learned_addr learned my addr 172.21.15.103:0/4083074569 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:54.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.120+0000 7f75bf7fe700 1 -- 172.21.15.103:0/4083074569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c01c6c10 con 0x7f75c009d8a0 2023-07-02T08:59:54.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.120+0000 7f75bf7fe700 1 --2- 172.21.15.103:0/4083074569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c01c7250 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.122+0000 7f75a7fff700 1 -- 172.21.15.103:0/4083074569 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f75b0013740 con 0x7f75c009d8a0 2023-07-02T08:59:54.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.122+0000 7f75a7fff700 1 -- 172.21.15.103:0/4083074569 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f75b00138a0 con 0x7f75c009d8a0 2023-07-02T08:59:54.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.122+0000 7f75a7fff700 1 -- 172.21.15.103:0/4083074569 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f75b0013b40 con 0x7f75c009d8a0 2023-07-02T08:59:54.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.122+0000 7f934affd700 1 -- 172.21.15.103:0/712287083 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f935412ec10 con 0x7f9354130b20 2023-07-02T08:59:54.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.122+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c01c6eb0 con 0x7f75c009d8a0 2023-07-02T08:59:54.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.122+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c01a9100 con 0x7f75c009d8a0 2023-07-02T08:59:54.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.124+0000 7f75a7fff700 1 -- 172.21.15.103:0/4083074569 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f75b0013cf0 con 0x7f75c009d8a0 2023-07-02T08:59:54.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.124+0000 7f75a7fff700 1 --2- 172.21.15.103:0/4083074569 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75ac0377c0 0x7f75ac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:54.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.124+0000 7f75a7fff700 1 -- 172.21.15.103:0/4083074569 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f75b0018040 con 0x7f75c009d8a0 2023-07-02T08:59:54.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.124+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c009b250 con 0x7f75c009d8a0 2023-07-02T08:59:54.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.124+0000 7f75beffd700 1 --2- 172.21.15.103:0/4083074569 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75ac0377c0 0x7f75ac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:54.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.125+0000 7f75beffd700 1 --2- 172.21.15.103:0/4083074569 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75ac0377c0 0x7f75ac039cd0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:54.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.129+0000 7f75a7fff700 1 -- 172.21.15.103:0/4083074569 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f75b0011380 con 0x7f75c009d8a0 2023-07-02T08:59:54.293 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.292+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa59411d030 con 0x7fa59409d8a0 2023-07-02T08:59:54.293 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.292+0000 7fa590ff9700 1 -- 172.21.15.103:0/2759182098 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (crc 0 0 0) 0x7fa58400ddf0 con 0x7fa59409d8a0 2023-07-02T08:59:54.294 INFO:teuthology.orchestra.run.smithi103.stdout:38654705680 2023-07-02T08:59:54.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5800377c0 msgr2=0x7fa580039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 --2- 172.21.15.103:0/2759182098 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5800377c0 0x7fa580039cd0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409d8a0 msgr2=0x7fa5941c73b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 --2- 172.21.15.103:0/2759182098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409d8a0 0x7fa5941c73b0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 shutdown_connections 2023-07-02T08:59:54.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 --2- 172.21.15.103:0/2759182098 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5800377c0 0x7fa580039cd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 --2- 172.21.15.103:0/2759182098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa59409d8a0 0x7fa5941c73b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 >> 172.21.15.103:0/2759182098 conn(0x7fa5940a3280 msgr2=0x7fa5941348a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:54.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 shutdown_connections 2023-07-02T08:59:54.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.294+0000 7fa59aa09700 1 -- 172.21.15.103:0/2759182098 wait complete. 2023-07-02T08:59:54.309 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705680 for osd.0 2023-07-02T08:59:54.309 DEBUG:teuthology.parallel:result is None 2023-07-02T08:59:54.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.317+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f23f00a1920 con 0x7f23f0132f50 2023-07-02T08:59:54.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.317+0000 7f23e5ffb700 1 -- 172.21.15.103:0/4191621723 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f23ec00a120 con 0x7f23f0132f50 2023-07-02T08:59:54.319 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:54.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.319+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9354066dd0 con 0x7f9354130b20 2023-07-02T08:59:54.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.319+0000 7f934affd700 1 -- 172.21.15.103:0/712287083 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (crc 0 0 0) 0x7f9354066dd0 con 0x7f9354130b20 2023-07-02T08:59:54.321 INFO:teuthology.orchestra.run.smithi103.stdout:38654705680 2023-07-02T08:59:54.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23d40377c0 msgr2=0x7f23d4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 --2- 172.21.15.103:0/4191621723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23d40377c0 0x7f23d4039cd0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0132f50 msgr2=0x7f23f012c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 --2- 172.21.15.103:0/4191621723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0132f50 0x7f23f012c270 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 shutdown_connections 2023-07-02T08:59:54.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 --2- 172.21.15.103:0/4191621723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f23d40377c0 0x7f23d4039cd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 --2- 172.21.15.103:0/4191621723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23f0132f50 0x7f23f012c270 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 >> 172.21.15.103:0/4191621723 conn(0x7f23f00a2080 msgr2=0x7f23f01bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:54.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 shutdown_connections 2023-07-02T08:59:54.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.320+0000 7f23f7a13700 1 -- 172.21.15.103:0/4191621723 wait complete. 2023-07-02T08:59:54.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.321+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f93380377c0 msgr2=0x7f9338039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.321+0000 7f935c719700 1 --2- 172.21.15.103:0/712287083 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f93380377c0 0x7f9338039cd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.321+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354130b20 msgr2=0x7f93541d3d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.321+0000 7f935c719700 1 --2- 172.21.15.103:0/712287083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354130b20 0x7f93541d3d70 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.322+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 shutdown_connections 2023-07-02T08:59:54.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.322+0000 7f935c719700 1 --2- 172.21.15.103:0/712287083 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f93380377c0 0x7f9338039cd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.322+0000 7f935c719700 1 --2- 172.21.15.103:0/712287083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9354130b20 0x7f93541d3d70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.322+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 >> 172.21.15.103:0/712287083 conn(0x7f935409f500 msgr2=0x7f9354131980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:54.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.322+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 shutdown_connections 2023-07-02T08:59:54.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.322+0000 7f935c719700 1 -- 172.21.15.103:0/712287083 wait complete. 2023-07-02T08:59:54.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.327+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f75c01c70a0 con 0x7f75c009d8a0 2023-07-02T08:59:54.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.327+0000 7f75a7fff700 1 -- 172.21.15.103:0/4083074569 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (crc 0 0 0) 0x7f75b000f2a0 con 0x7f75c009d8a0 2023-07-02T08:59:54.329 INFO:teuthology.orchestra.run.smithi103.stdout:38654705680 2023-07-02T08:59:54.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75ac0377c0 msgr2=0x7f75ac039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 --2- 172.21.15.103:0/4083074569 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75ac0377c0 0x7f75ac039cd0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 msgr2=0x7f75c01c7250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:54.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 --2- 172.21.15.103:0/4083074569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c01c7250 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 shutdown_connections 2023-07-02T08:59:54.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 --2- 172.21.15.103:0/4083074569 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75ac0377c0 0x7f75ac039cd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 --2- 172.21.15.103:0/4083074569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75c009d8a0 0x7f75c01c7250 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:54.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 >> 172.21.15.103:0/4083074569 conn(0x7f75c00a3280 msgr2=0x7f75c019e200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:54.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 shutdown_connections 2023-07-02T08:59:54.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:54.329+0000 7f75c5bee700 1 -- 172.21.15.103:0/4083074569 wait complete. 2023-07-02T08:59:54.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:54.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705680 for osd.2 2023-07-02T08:59:54.340 DEBUG:teuthology.parallel:result is None 2023-07-02T08:59:54.347 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705680 for osd.3 2023-07-02T08:59:54.347 DEBUG:teuthology.parallel:result is None 2023-07-02T08:59:55.340 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:55.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.478+0000 7f477559f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:55.484 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.483+0000 7f477559f700 1 Processor -- start 2023-07-02T08:59:55.484 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.483+0000 7f477559f700 1 -- start start 2023-07-02T08:59:55.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.483+0000 7f477559f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f477012f250 0x7f477012f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:55.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f477559f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477006df60 con 0x7f4770098a00 2023-07-02T08:59:55.486 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f477559f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477009a710 con 0x7f477012f250 2023-07-02T08:59:55.486 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f477012f250 0x7f477012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:55.486 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476e7fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4770098a00 0x7f4770098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37014/0 (socket says 172.21.15.103:37014) 2023-07-02T08:59:55.486 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476e7fc700 1 -- 172.21.15.103:0/3320707581 learned_addr learned my addr 172.21.15.103:0/3320707581 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:55.486 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476effd700 1 -- 172.21.15.103:0/3320707581 >> v1:172.21.15.103:6789/0 conn(0x7f4770098a00 legacy=0x7f4770098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:55.486 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476effd700 1 -- 172.21.15.103:0/3320707581 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477012ed20 con 0x7f477012f250 2023-07-02T08:59:55.486 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476effd700 1 --2- 172.21.15.103:0/3320707581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f477012f250 0x7f477012f660 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83adec2810235d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:55.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476dffb700 1 -- 172.21.15.103:0/3320707581 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f476400d450 con 0x7f477012f250 2023-07-02T08:59:55.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476dffb700 1 -- 172.21.15.103:0/3320707581 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f476400d5b0 con 0x7f477012f250 2023-07-02T08:59:55.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.484+0000 7f476dffb700 1 -- 172.21.15.103:0/3320707581 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f476400d850 con 0x7f477012f250 2023-07-02T08:59:55.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.485+0000 7f477559f700 1 -- 172.21.15.103:0/3320707581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f477012f250 msgr2=0x7f477012f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:55.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.485+0000 7f477559f700 1 --2- 172.21.15.103:0/3320707581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f477012f250 0x7f477012f660 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:55.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.485+0000 7f477559f700 1 -- 172.21.15.103:0/3320707581 shutdown_connections 2023-07-02T08:59:55.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.485+0000 7f477559f700 1 --2- 172.21.15.103:0/3320707581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f477012f250 0x7f477012f660 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:55.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.485+0000 7f477559f700 1 -- 172.21.15.103:0/3320707581 >> 172.21.15.103:0/3320707581 conn(0x7f477009d810 msgr2=0x7f477009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:55.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.485+0000 7f477559f700 1 -- 172.21.15.103:0/3320707581 shutdown_connections 2023-07-02T08:59:55.489 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.485+0000 7f477559f700 1 -- 172.21.15.103:0/3320707581 wait complete. 2023-07-02T08:59:55.489 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.485+0000 7f477559f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:55.489 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.486+0000 7f477559f700 1 Processor -- start 2023-07-02T08:59:55.489 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.486+0000 7f477559f700 1 -- start start 2023-07-02T08:59:55.490 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.487+0000 7f477559f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4770098a00 0x7f4770117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:55.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.487+0000 7f477559f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477011a9f0 con 0x7f4770098a00 2023-07-02T08:59:55.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.487+0000 7f476effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4770098a00 0x7f4770117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:55.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.487+0000 7f476effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4770098a00 0x7f4770117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43750/0 (socket says 172.21.15.103:43750) 2023-07-02T08:59:55.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.487+0000 7f476effd700 1 -- 172.21.15.103:0/789752460 learned_addr learned my addr 172.21.15.103:0/789752460 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:55.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.487+0000 7f476effd700 1 -- 172.21.15.103:0/789752460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770118f50 con 0x7f4770098a00 2023-07-02T08:59:55.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.487+0000 7f476effd700 1 --2- 172.21.15.103:0/789752460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4770098a00 0x7f4770117360 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:55.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.488+0000 7f47577fe700 1 -- 172.21.15.103:0/789752460 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f476400c990 con 0x7f4770098a00 2023-07-02T08:59:55.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.488+0000 7f47577fe700 1 -- 172.21.15.103:0/789752460 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f476400caf0 con 0x7f4770098a00 2023-07-02T08:59:55.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.488+0000 7f47577fe700 1 -- 172.21.15.103:0/789752460 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f476400cd90 con 0x7f4770098a00 2023-07-02T08:59:55.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.488+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477011a650 con 0x7f4770098a00 2023-07-02T08:59:55.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.488+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477011afe0 con 0x7f4770098a00 2023-07-02T08:59:55.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.489+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477011a840 con 0x7f4770098a00 2023-07-02T08:59:55.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.489+0000 7f47577fe700 1 -- 172.21.15.103:0/789752460 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4764015040 con 0x7f4770098a00 2023-07-02T08:59:55.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.489+0000 7f47577fe700 1 --2- 172.21.15.103:0/789752460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f475c0377c0 0x7f475c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:55.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.489+0000 7f47577fe700 1 -- 172.21.15.103:0/789752460 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4764012040 con 0x7f4770098a00 2023-07-02T08:59:55.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.489+0000 7f476e7fc700 1 --2- 172.21.15.103:0/789752460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f475c0377c0 0x7f475c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:55.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.490+0000 7f476e7fc700 1 --2- 172.21.15.103:0/789752460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f475c0377c0 0x7f475c039cd0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:55.494 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.493+0000 7f47577fe700 1 -- 172.21.15.103:0/789752460 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f4764014c50 con 0x7f4770098a00 2023-07-02T08:59:55.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.685+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f47701198a0 con 0x7f4770098a00 2023-07-02T08:59:55.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.685+0000 7f47577fe700 1 -- 172.21.15.103:0/789752460 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4764014a70 con 0x7f4770098a00 2023-07-02T08:59:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:55.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.687+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f475c0377c0 msgr2=0x7f475c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:55.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.687+0000 7f477559f700 1 --2- 172.21.15.103:0/789752460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f475c0377c0 0x7f475c039cd0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:55.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.687+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4770098a00 msgr2=0x7f4770117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:55.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.687+0000 7f477559f700 1 --2- 172.21.15.103:0/789752460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4770098a00 0x7f4770117360 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:55.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.688+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 shutdown_connections 2023-07-02T08:59:55.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.688+0000 7f477559f700 1 --2- 172.21.15.103:0/789752460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f475c0377c0 0x7f475c039cd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:55.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.688+0000 7f477559f700 1 --2- 172.21.15.103:0/789752460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4770098a00 0x7f4770117360 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:55.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.688+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 >> 172.21.15.103:0/789752460 conn(0x7f477009d810 msgr2=0x7f477009f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:55.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.688+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 shutdown_connections 2023-07-02T08:59:55.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:55.688+0000 7f477559f700 1 -- 172.21.15.103:0/789752460 wait complete. 2023-07-02T08:59:55.702 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:56.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:56.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.841+0000 7f790c69e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:56.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.846+0000 7f790c69e700 1 Processor -- start 2023-07-02T08:59:56.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.846+0000 7f790c69e700 1 -- start start 2023-07-02T08:59:56.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.847+0000 7f790c69e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7904132f50 0x7f7904133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:56.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.847+0000 7f790c69e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790409e260 con 0x7f790409c580 2023-07-02T08:59:56.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.847+0000 7f790c69e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79040a1640 con 0x7f7904132f50 2023-07-02T08:59:56.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.847+0000 7f790a43a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7904132f50 0x7f7904133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:56.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.847+0000 7f7909c39700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f790409c580 0x7f790409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37016/0 (socket says 172.21.15.103:37016) 2023-07-02T08:59:56.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.847+0000 7f7909c39700 1 -- 172.21.15.103:0/682040710 learned_addr learned my addr 172.21.15.103:0/682040710 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:56.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.847+0000 7f790a43a700 1 -- 172.21.15.103:0/682040710 >> v1:172.21.15.103:6789/0 conn(0x7f790409c580 legacy=0x7f790409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:56.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.847+0000 7f790a43a700 1 -- 172.21.15.103:0/682040710 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904132a20 con 0x7f7904132f50 2023-07-02T08:59:56.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.848+0000 7f790a43a700 1 --2- 172.21.15.103:0/682040710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7904132f50 0x7f7904133360 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf3e4d8369ec6b6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:56.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.848+0000 7f7909438700 1 -- 172.21.15.103:0/682040710 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f78f4009cd0 con 0x7f7904132f50 2023-07-02T08:59:56.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.848+0000 7f7909438700 1 -- 172.21.15.103:0/682040710 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f78f4009e30 con 0x7f7904132f50 2023-07-02T08:59:56.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.848+0000 7f7909438700 1 -- 172.21.15.103:0/682040710 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f78f400e360 con 0x7f7904132f50 2023-07-02T08:59:56.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.848+0000 7f790c69e700 1 -- 172.21.15.103:0/682040710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7904132f50 msgr2=0x7f7904133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:56.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.848+0000 7f790c69e700 1 --2- 172.21.15.103:0/682040710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7904132f50 0x7f7904133360 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:56.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.849+0000 7f790c69e700 1 -- 172.21.15.103:0/682040710 shutdown_connections 2023-07-02T08:59:56.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.849+0000 7f790c69e700 1 --2- 172.21.15.103:0/682040710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7904132f50 0x7f7904133360 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:56.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.849+0000 7f790c69e700 1 -- 172.21.15.103:0/682040710 >> 172.21.15.103:0/682040710 conn(0x7f79040a2080 msgr2=0x7f79040a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:56.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.849+0000 7f790c69e700 1 -- 172.21.15.103:0/682040710 shutdown_connections 2023-07-02T08:59:56.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.849+0000 7f790c69e700 1 -- 172.21.15.103:0/682040710 wait complete. 2023-07-02T08:59:56.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.849+0000 7f790c69e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:56.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.850+0000 7f790c69e700 1 Processor -- start 2023-07-02T08:59:56.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.850+0000 7f790c69e700 1 -- start start 2023-07-02T08:59:56.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.850+0000 7f790c69e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f790409c580 0x7f790412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:56.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.850+0000 7f790c69e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790406f9d0 con 0x7f790409c580 2023-07-02T08:59:56.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f790a43a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f790409c580 0x7f790412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:56.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f790a43a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f790409c580 0x7f790412b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43768/0 (socket says 172.21.15.103:43768) 2023-07-02T08:59:56.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f790a43a700 1 -- 172.21.15.103:0/3546896981 learned_addr learned my addr 172.21.15.103:0/3546896981 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:56.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f790a43a700 1 -- 172.21.15.103:0/3546896981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904133a00 con 0x7f790409c580 2023-07-02T08:59:56.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f790a43a700 1 --2- 172.21.15.103:0/3546896981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f790409c580 0x7f790412b7b0 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:56.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f78faffd700 1 -- 172.21.15.103:0/3546896981 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f78f4013530 con 0x7f790409c580 2023-07-02T08:59:56.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f78faffd700 1 -- 172.21.15.103:0/3546896981 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f78f4013690 con 0x7f790409c580 2023-07-02T08:59:56.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f78faffd700 1 -- 172.21.15.103:0/3546896981 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f78f4013930 con 0x7f790409c580 2023-07-02T08:59:56.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790406d160 con 0x7f790409c580 2023-07-02T08:59:56.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.851+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790406da70 con 0x7f790409c580 2023-07-02T08:59:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.852+0000 7f78faffd700 1 -- 172.21.15.103:0/3546896981 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f78f4013ae0 con 0x7f790409c580 2023-07-02T08:59:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.852+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7904061ea0 con 0x7f790409c580 2023-07-02T08:59:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.852+0000 7f78faffd700 1 --2- 172.21.15.103:0/3546896981 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78e80377c0 0x7f78e8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.852+0000 7f78faffd700 1 -- 172.21.15.103:0/3546896981 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f78f4018040 con 0x7f790409c580 2023-07-02T08:59:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.853+0000 7f7909c39700 1 --2- 172.21.15.103:0/3546896981 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78e80377c0 0x7f78e8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.853+0000 7f7909c39700 1 --2- 172.21.15.103:0/3546896981 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78e80377c0 0x7f78e8039cd0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:56.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:56.857+0000 7f78faffd700 1 -- 172.21.15.103:0/3546896981 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f78f4018dd0 con 0x7f790409c580 2023-07-02T08:59:56.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2023-07-02T08:59:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.046+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f79040725b0 con 0x7f790409c580 2023-07-02T08:59:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.046+0000 7f78faffd700 1 -- 172.21.15.103:0/3546896981 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f78f40189f0 con 0x7f790409c580 2023-07-02T08:59:57.048 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:57.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78e80377c0 msgr2=0x7f78e8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:57.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 --2- 172.21.15.103:0/3546896981 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78e80377c0 0x7f78e8039cd0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:57.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f790409c580 msgr2=0x7f790412b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:57.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 --2- 172.21.15.103:0/3546896981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f790409c580 0x7f790412b7b0 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:57.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 shutdown_connections 2023-07-02T08:59:57.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 --2- 172.21.15.103:0/3546896981 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78e80377c0 0x7f78e8039cd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:57.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 --2- 172.21.15.103:0/3546896981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f790409c580 0x7f790412b7b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:57.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 >> 172.21.15.103:0/3546896981 conn(0x7f79040a2080 msgr2=0x7f79041babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:57.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 shutdown_connections 2023-07-02T08:59:57.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:57.048+0000 7f790c69e700 1 -- 172.21.15.103:0/3546896981 wait complete. 2023-07-02T08:59:57.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:58.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:58.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.202+0000 7fbd85e8b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:58.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.207+0000 7fbd85e8b700 1 Processor -- start 2023-07-02T08:59:58.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.207+0000 7fbd85e8b700 1 -- start start 2023-07-02T08:59:58.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd85e8b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbd80132f50 0x7fbd80133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:58.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd85e8b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8009e260 con 0x7fbd8009c580 2023-07-02T08:59:58.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd85e8b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd800a1640 con 0x7fbd80132f50 2023-07-02T08:59:58.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd7f7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbd80132f50 0x7fbd80133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:58.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd7effd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fbd8009c580 0x7fbd8009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37024/0 (socket says 172.21.15.103:37024) 2023-07-02T08:59:58.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd7f7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbd80132f50 0x7fbd80133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43776/0 (socket says 172.21.15.103:43776) 2023-07-02T08:59:58.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd7effd700 1 -- 172.21.15.103:0/1314653370 learned_addr learned my addr 172.21.15.103:0/1314653370 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:58.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd7f7fe700 1 -- 172.21.15.103:0/1314653370 >> v1:172.21.15.103:6789/0 conn(0x7fbd8009c580 legacy=0x7fbd8009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:58.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.208+0000 7fbd7f7fe700 1 -- 172.21.15.103:0/1314653370 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80132a20 con 0x7fbd80132f50 2023-07-02T08:59:58.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd7f7fe700 1 --2- 172.21.15.103:0/1314653370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd80132f50 0x7fbd80133360 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835809f6048e1561 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:58.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd7e7fc700 1 -- 172.21.15.103:0/1314653370 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fbd74009cd0 con 0x7fbd80132f50 2023-07-02T08:59:58.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd7e7fc700 1 -- 172.21.15.103:0/1314653370 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fbd74009e30 con 0x7fbd80132f50 2023-07-02T08:59:58.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd7e7fc700 1 -- 172.21.15.103:0/1314653370 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fbd7400e360 con 0x7fbd80132f50 2023-07-02T08:59:58.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd85e8b700 1 -- 172.21.15.103:0/1314653370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd80132f50 msgr2=0x7fbd80133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:58.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd85e8b700 1 --2- 172.21.15.103:0/1314653370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd80132f50 0x7fbd80133360 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:58.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd85e8b700 1 -- 172.21.15.103:0/1314653370 shutdown_connections 2023-07-02T08:59:58.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd85e8b700 1 --2- 172.21.15.103:0/1314653370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd80132f50 0x7fbd80133360 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:58.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd85e8b700 1 -- 172.21.15.103:0/1314653370 >> 172.21.15.103:0/1314653370 conn(0x7fbd800a2080 msgr2=0x7fbd800a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:58.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.209+0000 7fbd85e8b700 1 -- 172.21.15.103:0/1314653370 shutdown_connections 2023-07-02T08:59:58.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.210+0000 7fbd85e8b700 1 -- 172.21.15.103:0/1314653370 wait complete. 2023-07-02T08:59:58.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.210+0000 7fbd85e8b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:58.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.211+0000 7fbd85e8b700 1 Processor -- start 2023-07-02T08:59:58.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.211+0000 7fbd85e8b700 1 -- start start 2023-07-02T08:59:58.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.212+0000 7fbd85e8b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd8009c580 0x7fbd8012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.212+0000 7fbd85e8b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8006f9d0 con 0x7fbd8009c580 2023-07-02T08:59:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.212+0000 7fbd7f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd8009c580 0x7fbd8012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.212+0000 7fbd7f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd8009c580 0x7fbd8012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43788/0 (socket says 172.21.15.103:43788) 2023-07-02T08:59:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.212+0000 7fbd7f7fe700 1 -- 172.21.15.103:0/2033281038 learned_addr learned my addr 172.21.15.103:0/2033281038 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.212+0000 7fbd7f7fe700 1 -- 172.21.15.103:0/2033281038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80133a00 con 0x7fbd8009c580 2023-07-02T08:59:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.212+0000 7fbd7f7fe700 1 --2- 172.21.15.103:0/2033281038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd8009c580 0x7fbd8012b7b0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.213+0000 7fbd67fff700 1 -- 172.21.15.103:0/2033281038 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fbd74013530 con 0x7fbd8009c580 2023-07-02T08:59:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.213+0000 7fbd67fff700 1 -- 172.21.15.103:0/2033281038 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fbd74013690 con 0x7fbd8009c580 2023-07-02T08:59:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.213+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8006d160 con 0x7fbd8009c580 2023-07-02T08:59:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.213+0000 7fbd67fff700 1 -- 172.21.15.103:0/2033281038 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fbd74013930 con 0x7fbd8009c580 2023-07-02T08:59:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.213+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8006da70 con 0x7fbd8009c580 2023-07-02T08:59:58.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.213+0000 7fbd67fff700 1 -- 172.21.15.103:0/2033281038 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fbd74013ae0 con 0x7fbd8009c580 2023-07-02T08:59:58.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.213+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd80061ea0 con 0x7fbd8009c580 2023-07-02T08:59:58.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.214+0000 7fbd67fff700 1 --2- 172.21.15.103:0/2033281038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd6c0377c0 0x7fbd6c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:58.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.214+0000 7fbd67fff700 1 -- 172.21.15.103:0/2033281038 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fbd74018040 con 0x7fbd8009c580 2023-07-02T08:59:58.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.214+0000 7fbd7effd700 1 --2- 172.21.15.103:0/2033281038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd6c0377c0 0x7fbd6c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:58.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.214+0000 7fbd7effd700 1 --2- 172.21.15.103:0/2033281038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd6c0377c0 0x7fbd6c039cd0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:58.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.219+0000 7fbd67fff700 1 -- 172.21.15.103:0/2033281038 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fbd74018e00 con 0x7fbd8009c580 2023-07-02T08:59:58.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.413+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbd800725b0 con 0x7fbd8009c580 2023-07-02T08:59:58.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.413+0000 7fbd67fff700 1 -- 172.21.15.103:0/2033281038 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fbd74010e60 con 0x7fbd8009c580 2023-07-02T08:59:58.415 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:58.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd6c0377c0 msgr2=0x7fbd6c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:58.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 --2- 172.21.15.103:0/2033281038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd6c0377c0 0x7fbd6c039cd0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:58.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd8009c580 msgr2=0x7fbd8012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 --2- 172.21.15.103:0/2033281038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd8009c580 0x7fbd8012b7b0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 shutdown_connections 2023-07-02T08:59:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 --2- 172.21.15.103:0/2033281038 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd6c0377c0 0x7fbd6c039cd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 --2- 172.21.15.103:0/2033281038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd8009c580 0x7fbd8012b7b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 >> 172.21.15.103:0/2033281038 conn(0x7fbd800a2080 msgr2=0x7fbd801babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 shutdown_connections 2023-07-02T08:59:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:58.416+0000 7fbd85e8b700 1 -- 172.21.15.103:0/2033281038 wait complete. 2023-07-02T08:59:58.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T08:59:59.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T08:59:59.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.570+0000 7f1ec7efb700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:59.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.575+0000 7f1ec7efb700 1 Processor -- start 2023-07-02T08:59:59.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.575+0000 7f1ec7efb700 1 -- start start 2023-07-02T08:59:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.575+0000 7f1ec7efb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ec0133360 0x7f1ec009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.575+0000 7f1ec7efb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec009e260 con 0x7f1ec0132f50 2023-07-02T08:59:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.575+0000 7f1ec7efb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec00a1640 con 0x7f1ec0133360 2023-07-02T08:59:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec5c97700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1ec0132f50 0x7f1ec009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37038/0 (socket says 172.21.15.103:37038) 2023-07-02T08:59:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec5496700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ec0133360 0x7f1ec009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec5c97700 1 -- 172.21.15.103:0/678765156 learned_addr learned my addr 172.21.15.103:0/678765156 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T08:59:59.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec5496700 1 -- 172.21.15.103:0/678765156 >> v1:172.21.15.103:6789/0 conn(0x7f1ec0132f50 legacy=0x7f1ec009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:59.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec5496700 1 -- 172.21.15.103:0/678765156 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0132a20 con 0x7f1ec0133360 2023-07-02T08:59:59.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec5496700 1 --2- 172.21.15.103:0/678765156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0133360 0x7f1ec009cc90 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a268bf200c175308 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:59.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec4c95700 1 -- 172.21.15.103:0/678765156 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1ebc009cd0 con 0x7f1ec0133360 2023-07-02T08:59:59.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec4c95700 1 -- 172.21.15.103:0/678765156 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1ebc009e30 con 0x7f1ec0133360 2023-07-02T08:59:59.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.576+0000 7f1ec4c95700 1 -- 172.21.15.103:0/678765156 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1ebc00e360 con 0x7f1ec0133360 2023-07-02T08:59:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.577+0000 7f1ec7efb700 1 -- 172.21.15.103:0/678765156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0133360 msgr2=0x7f1ec009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.577+0000 7f1ec7efb700 1 --2- 172.21.15.103:0/678765156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0133360 0x7f1ec009cc90 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.577+0000 7f1ec7efb700 1 -- 172.21.15.103:0/678765156 shutdown_connections 2023-07-02T08:59:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.577+0000 7f1ec7efb700 1 --2- 172.21.15.103:0/678765156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0133360 0x7f1ec009cc90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.577+0000 7f1ec7efb700 1 -- 172.21.15.103:0/678765156 >> 172.21.15.103:0/678765156 conn(0x7f1ec00a2080 msgr2=0x7f1ec00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.577+0000 7f1ec7efb700 1 -- 172.21.15.103:0/678765156 shutdown_connections 2023-07-02T08:59:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.577+0000 7f1ec7efb700 1 -- 172.21.15.103:0/678765156 wait complete. 2023-07-02T08:59:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.577+0000 7f1ec7efb700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T08:59:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.578+0000 7f1ec7efb700 1 Processor -- start 2023-07-02T08:59:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.578+0000 7f1ec7efb700 1 -- start start 2023-07-02T08:59:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec7efb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0132f50 0x7f1ec012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec7efb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec006d260 con 0x7f1ec0132f50 2023-07-02T08:59:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec5c97700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0132f50 0x7f1ec012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec5c97700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0132f50 0x7f1ec012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43804/0 (socket says 172.21.15.103:43804) 2023-07-02T08:59:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec5c97700 1 -- 172.21.15.103:0/1274948359 learned_addr learned my addr 172.21.15.103:0/1274948359 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T08:59:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec5c97700 1 -- 172.21.15.103:0/1274948359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec012bb90 con 0x7f1ec0132f50 2023-07-02T08:59:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec5c97700 1 --2- 172.21.15.103:0/1274948359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0132f50 0x7f1ec012bea0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1eb67fc700 1 -- 172.21.15.103:0/1274948359 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1eb0005a60 con 0x7f1ec0132f50 2023-07-02T08:59:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1eb67fc700 1 -- 172.21.15.103:0/1274948359 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1eb0005bc0 con 0x7f1ec0132f50 2023-07-02T08:59:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec006f730 con 0x7f1ec0132f50 2023-07-02T08:59:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1eb67fc700 1 -- 172.21.15.103:0/1274948359 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1eb0005e60 con 0x7f1ec0132f50 2023-07-02T08:59:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.579+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec006f490 con 0x7f1ec0132f50 2023-07-02T08:59:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.580+0000 7f1eb67fc700 1 -- 172.21.15.103:0/1274948359 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f1eb00070b0 con 0x7f1ec0132f50 2023-07-02T08:59:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.580+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec0061ea0 con 0x7f1ec0132f50 2023-07-02T08:59:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.580+0000 7f1eb67fc700 1 --2- 172.21.15.103:0/1274948359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1ea40377c0 0x7f1ea4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T08:59:59.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.580+0000 7f1eb67fc700 1 -- 172.21.15.103:0/1274948359 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f1eb0046ab0 con 0x7f1ec0132f50 2023-07-02T08:59:59.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.581+0000 7f1ec5496700 1 --2- 172.21.15.103:0/1274948359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1ea40377c0 0x7f1ea4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T08:59:59.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.581+0000 7f1ec5496700 1 --2- 172.21.15.103:0/1274948359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1ea40377c0 0x7f1ea4039cd0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T08:59:59.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.586+0000 7f1eb67fc700 1 -- 172.21.15.103:0/1274948359 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f1eb000a4a0 con 0x7f1ec0132f50 2023-07-02T08:59:59.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.776+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1ec006f490 con 0x7f1ec0132f50 2023-07-02T08:59:59.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.777+0000 7f1eb67fc700 1 -- 172.21.15.103:0/1274948359 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f1ec006f490 con 0x7f1ec0132f50 2023-07-02T08:59:59.778 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T08:59:59.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1ea40377c0 msgr2=0x7f1ea4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:59.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 --2- 172.21.15.103:0/1274948359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1ea40377c0 0x7f1ea4039cd0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:59.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0132f50 msgr2=0x7f1ec012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T08:59:59.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 --2- 172.21.15.103:0/1274948359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0132f50 0x7f1ec012bea0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:59.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 shutdown_connections 2023-07-02T08:59:59.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 --2- 172.21.15.103:0/1274948359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1ea40377c0 0x7f1ea4039cd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:59.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 --2- 172.21.15.103:0/1274948359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ec0132f50 0x7f1ec012bea0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T08:59:59.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 >> 172.21.15.103:0/1274948359 conn(0x7f1ec00a2080 msgr2=0x7f1ec01bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T08:59:59.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 shutdown_connections 2023-07-02T08:59:59.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T08:59:59.780+0000 7f1ec7efb700 1 -- 172.21.15.103:0/1274948359 wait complete. 2023-07-02T08:59:59.793 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:00.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:00.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.936+0000 7fc72ac01700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:00.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.942+0000 7fc72999f700 1 Processor -- start 2023-07-02T09:00:00.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.942+0000 7fc72999f700 1 -- start start 2023-07-02T09:00:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.942+0000 7fc72999f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc724131b10 0x7fc724133f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.942+0000 7fc72999f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72406c160 con 0x7fc72412ecf0 2023-07-02T09:00:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.942+0000 7fc72999f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72406c5a0 con 0x7fc724131b10 2023-07-02T09:00:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.942+0000 7fc723fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc724131b10 0x7fc724133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.942+0000 7fc72899d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc72412ecf0 0x7fc72412f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37050/0 (socket says 172.21.15.103:37050) 2023-07-02T09:00:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.943+0000 7fc72899d700 1 -- 172.21.15.103:0/3865208 learned_addr learned my addr 172.21.15.103:0/3865208 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.943+0000 7fc723fff700 1 -- 172.21.15.103:0/3865208 >> v1:172.21.15.103:6789/0 conn(0x7fc72412ecf0 legacy=0x7fc72412f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.943+0000 7fc723fff700 1 -- 172.21.15.103:0/3865208 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72412e8a0 con 0x7fc724131b10 2023-07-02T09:00:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.943+0000 7fc7237fe700 1 -- 172.21.15.103:0/3865208 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431381394 0 0) 0x7fc72406c160 con 0x7fc72412ecf0 2023-07-02T09:00:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.943+0000 7fc723fff700 1 --2- 172.21.15.103:0/3865208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc724131b10 0x7fc724133f30 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9a43cd0b4c3d78 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:00.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.943+0000 7fc7237fe700 1 -- 172.21.15.103:0/3865208 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc71400e040 con 0x7fc724131b10 2023-07-02T09:00:00.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.943+0000 7fc7237fe700 1 -- 172.21.15.103:0/3865208 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc714007670 con 0x7fc724131b10 2023-07-02T09:00:00.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.943+0000 7fc7237fe700 1 -- 172.21.15.103:0/3865208 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc714007910 con 0x7fc724131b10 2023-07-02T09:00:00.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.944+0000 7fc72999f700 1 -- 172.21.15.103:0/3865208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc724131b10 msgr2=0x7fc724133f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:00.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.944+0000 7fc72999f700 1 --2- 172.21.15.103:0/3865208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc724131b10 0x7fc724133f30 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.944+0000 7fc72999f700 1 -- 172.21.15.103:0/3865208 shutdown_connections 2023-07-02T09:00:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.944+0000 7fc72999f700 1 --2- 172.21.15.103:0/3865208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc724131b10 0x7fc724133f30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.944+0000 7fc72999f700 1 -- 172.21.15.103:0/3865208 >> 172.21.15.103:0/3865208 conn(0x7fc72409ea00 msgr2=0x7fc72409ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.944+0000 7fc72999f700 1 -- 172.21.15.103:0/3865208 shutdown_connections 2023-07-02T09:00:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.944+0000 7fc72999f700 1 -- 172.21.15.103:0/3865208 wait complete. 2023-07-02T09:00:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.944+0000 7fc72999f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.945+0000 7fc72999f700 1 Processor -- start 2023-07-02T09:00:00.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.945+0000 7fc72999f700 1 -- start start 2023-07-02T09:00:00.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.946+0000 7fc72999f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc72412ecf0 0x7fc72411c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:00.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.946+0000 7fc72999f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72411dde0 con 0x7fc72412ecf0 2023-07-02T09:00:00.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.946+0000 7fc72899d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc72412ecf0 0x7fc72411c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:00.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.946+0000 7fc72899d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc72412ecf0 0x7fc72411c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43826/0 (socket says 172.21.15.103:43826) 2023-07-02T09:00:00.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.946+0000 7fc72899d700 1 -- 172.21.15.103:0/1566464970 learned_addr learned my addr 172.21.15.103:0/1566464970 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:00.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.946+0000 7fc72899d700 1 -- 172.21.15.103:0/1566464970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72411f7f0 con 0x7fc72412ecf0 2023-07-02T09:00:00.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.946+0000 7fc72899d700 1 --2- 172.21.15.103:0/1566464970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc72412ecf0 0x7fc72411c160 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:00.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.947+0000 7fc7217fa700 1 -- 172.21.15.103:0/1566464970 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc70c004b80 con 0x7fc72412ecf0 2023-07-02T09:00:00.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.947+0000 7fc7217fa700 1 -- 172.21.15.103:0/1566464970 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc70c004ce0 con 0x7fc72412ecf0 2023-07-02T09:00:00.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.947+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72411f4e0 con 0x7fc72412ecf0 2023-07-02T09:00:00.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.947+0000 7fc7217fa700 1 -- 172.21.15.103:0/1566464970 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc70c00c520 con 0x7fc72412ecf0 2023-07-02T09:00:00.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.947+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72411fe00 con 0x7fc72412ecf0 2023-07-02T09:00:00.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.947+0000 7fc7217fa700 1 -- 172.21.15.103:0/1566464970 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fc70c00c680 con 0x7fc72412ecf0 2023-07-02T09:00:00.951 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.948+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc724061ea0 con 0x7fc72412ecf0 2023-07-02T09:00:00.951 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.948+0000 7fc7217fa700 1 --2- 172.21.15.103:0/1566464970 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc7040377c0 0x7fc704039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:00.951 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.948+0000 7fc7217fa700 1 -- 172.21.15.103:0/1566464970 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc70c0469c0 con 0x7fc72412ecf0 2023-07-02T09:00:00.951 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.948+0000 7fc723fff700 1 --2- 172.21.15.103:0/1566464970 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc7040377c0 0x7fc704039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:00.951 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.948+0000 7fc723fff700 1 --2- 172.21.15.103:0/1566464970 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc7040377c0 0x7fc704039cd0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:00.952 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:00.952+0000 7fc7217fa700 1 -- 172.21.15.103:0/1566464970 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc70c009c80 con 0x7fc72412ecf0 2023-07-02T09:00:01.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.146+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc724072420 con 0x7fc72412ecf0 2023-07-02T09:00:01.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.146+0000 7fc7217fa700 1 -- 172.21.15.103:0/1566464970 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fc70c00a120 con 0x7fc72412ecf0 2023-07-02T09:00:01.147 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:01.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc7040377c0 msgr2=0x7fc704039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:01.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 --2- 172.21.15.103:0/1566464970 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc7040377c0 0x7fc704039cd0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:01.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc72412ecf0 msgr2=0x7fc72411c160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:01.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 --2- 172.21.15.103:0/1566464970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc72412ecf0 0x7fc72411c160 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:01.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 shutdown_connections 2023-07-02T09:00:01.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 --2- 172.21.15.103:0/1566464970 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc7040377c0 0x7fc704039cd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:01.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 --2- 172.21.15.103:0/1566464970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc72412ecf0 0x7fc72411c160 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:01.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 >> 172.21.15.103:0/1566464970 conn(0x7fc72409ea00 msgr2=0x7fc72412dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:01.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 shutdown_connections 2023-07-02T09:00:01.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:01.148+0000 7fc72999f700 1 -- 172.21.15.103:0/1566464970 wait complete. 2023-07-02T09:00:01.162 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:02.163 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.303+0000 7f15b699a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:02.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.308+0000 7f15b699a700 1 Processor -- start 2023-07-02T09:00:02.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.308+0000 7f15b699a700 1 -- start start 2023-07-02T09:00:02.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.308+0000 7f15b699a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f15b006f720 0x7f15b006fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:02.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.308+0000 7f15b699a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b0067780 con 0x7f15b0070070 2023-07-02T09:00:02.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.308+0000 7f15b699a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b0067f80 con 0x7f15b006f720 2023-07-02T09:00:02.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15affff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f15b006f720 0x7f15b006fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:02.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15affff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f15b006f720 0x7f15b006fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43838/0 (socket says 172.21.15.103:43838) 2023-07-02T09:00:02.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15affff700 1 -- 172.21.15.103:0/2366782853 learned_addr learned my addr 172.21.15.103:0/2366782853 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:02.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15af7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f15b0070070 0x7f15b01307f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37056/0 (socket says 172.21.15.103:37056) 2023-07-02T09:00:02.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15affff700 1 -- 172.21.15.103:0/2366782853 >> v1:172.21.15.103:6789/0 conn(0x7f15b0070070 legacy=0x7f15b01307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:02.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15affff700 1 -- 172.21.15.103:0/2366782853 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b0093f30 con 0x7f15b006f720 2023-07-02T09:00:02.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15affff700 1 --2- 172.21.15.103:0/2366782853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b006fb30 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc94fd00408b3246 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:02.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15aeffd700 1 -- 172.21.15.103:0/2366782853 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f15a4009d90 con 0x7f15b006f720 2023-07-02T09:00:02.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15aeffd700 1 -- 172.21.15.103:0/2366782853 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f15a400e230 con 0x7f15b006f720 2023-07-02T09:00:02.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.309+0000 7f15aeffd700 1 -- 172.21.15.103:0/2366782853 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f15a400e4d0 con 0x7f15b006f720 2023-07-02T09:00:02.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.310+0000 7f15b699a700 1 -- 172.21.15.103:0/2366782853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 msgr2=0x7f15b006fb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:02.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.310+0000 7f15b699a700 1 --2- 172.21.15.103:0/2366782853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b006fb30 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:02.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.310+0000 7f15b699a700 1 -- 172.21.15.103:0/2366782853 shutdown_connections 2023-07-02T09:00:02.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.310+0000 7f15b699a700 1 --2- 172.21.15.103:0/2366782853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b006fb30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:02.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.310+0000 7f15b699a700 1 -- 172.21.15.103:0/2366782853 >> 172.21.15.103:0/2366782853 conn(0x7f15b00a22a0 msgr2=0x7f15b006d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:02.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.310+0000 7f15b699a700 1 -- 172.21.15.103:0/2366782853 shutdown_connections 2023-07-02T09:00:02.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.310+0000 7f15b699a700 1 -- 172.21.15.103:0/2366782853 wait complete. 2023-07-02T09:00:02.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.310+0000 7f15b699a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:02.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.311+0000 7f15b699a700 1 Processor -- start 2023-07-02T09:00:02.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.311+0000 7f15b699a700 1 -- start start 2023-07-02T09:00:02.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.312+0000 7f15b699a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b009cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:02.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.312+0000 7f15b699a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b009b790 con 0x7f15b006f720 2023-07-02T09:00:02.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.312+0000 7f15affff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b009cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:02.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.312+0000 7f15affff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b009cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43850/0 (socket says 172.21.15.103:43850) 2023-07-02T09:00:02.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.312+0000 7f15affff700 1 -- 172.21.15.103:0/1493648579 learned_addr learned my addr 172.21.15.103:0/1493648579 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:02.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.312+0000 7f15affff700 1 -- 172.21.15.103:0/1493648579 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b009b9c0 con 0x7f15b006f720 2023-07-02T09:00:02.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.312+0000 7f15affff700 1 --2- 172.21.15.103:0/1493648579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b009cfc0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:02.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.313+0000 7f15acff9700 1 -- 172.21.15.103:0/1493648579 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f15a4014540 con 0x7f15b006f720 2023-07-02T09:00:02.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.313+0000 7f15acff9700 1 -- 172.21.15.103:0/1493648579 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f15a40146a0 con 0x7f15b006f720 2023-07-02T09:00:02.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.313+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b009bf00 con 0x7f15b006f720 2023-07-02T09:00:02.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.313+0000 7f15acff9700 1 -- 172.21.15.103:0/1493648579 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f15a4014940 con 0x7f15b006f720 2023-07-02T09:00:02.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.313+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b009b110 con 0x7f15b006f720 2023-07-02T09:00:02.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.313+0000 7f15acff9700 1 -- 172.21.15.103:0/1493648579 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f15a4014af0 con 0x7f15b006f720 2023-07-02T09:00:02.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.314+0000 7f15acff9700 1 --2- 172.21.15.103:0/1493648579 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f159c0377c0 0x7f159c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:02.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.314+0000 7f15acff9700 1 -- 172.21.15.103:0/1493648579 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f15a40482d0 con 0x7f15b006f720 2023-07-02T09:00:02.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.314+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b0061ea0 con 0x7f15b006f720 2023-07-02T09:00:02.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.314+0000 7f15af7fe700 1 --2- 172.21.15.103:0/1493648579 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f159c0377c0 0x7f159c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:02.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.314+0000 7f15af7fe700 1 --2- 172.21.15.103:0/1493648579 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f159c0377c0 0x7f159c039cd0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:02.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.319+0000 7f15acff9700 1 -- 172.21.15.103:0/1493648579 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f15a400d1c0 con 0x7f15b006f720 2023-07-02T09:00:02.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~17s 2023-07-02T09:00:02.510 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.510+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f15b0073ca0 con 0x7f15b006f720 2023-07-02T09:00:02.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.510+0000 7f15acff9700 1 -- 172.21.15.103:0/1493648579 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f15a4011320 con 0x7f15b006f720 2023-07-02T09:00:02.511 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:02.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f159c0377c0 msgr2=0x7f159c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:02.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 --2- 172.21.15.103:0/1493648579 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f159c0377c0 0x7f159c039cd0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:02.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 msgr2=0x7f15b009cfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:02.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 --2- 172.21.15.103:0/1493648579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b009cfc0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:02.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 shutdown_connections 2023-07-02T09:00:02.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 --2- 172.21.15.103:0/1493648579 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f159c0377c0 0x7f159c039cd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:02.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 --2- 172.21.15.103:0/1493648579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b006f720 0x7f15b009cfc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:02.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 >> 172.21.15.103:0/1493648579 conn(0x7f15b00a22a0 msgr2=0x7f15b008f840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:02.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 shutdown_connections 2023-07-02T09:00:02.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:02.513+0000 7f15b699a700 1 -- 172.21.15.103:0/1493648579 wait complete. 2023-07-02T09:00:02.526 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:03.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:03.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.667+0000 7f3c401de700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:03.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.672+0000 7f3c401de700 1 Processor -- start 2023-07-02T09:00:03.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.672+0000 7f3c401de700 1 -- start start 2023-07-02T09:00:03.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c401de700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c38133360 0x7f3c3809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:03.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c401de700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3809e260 con 0x7f3c38132f50 2023-07-02T09:00:03.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c401de700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c380a1640 con 0x7f3c38133360 2023-07-02T09:00:03.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3d779700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c38133360 0x7f3c3809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:03.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3df7a700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3c38132f50 0x7f3c3809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43188/0 (socket says 172.21.15.103:43188) 2023-07-02T09:00:03.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3df7a700 1 -- 172.21.15.103:0/982166462 learned_addr learned my addr 172.21.15.103:0/982166462 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:03.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3d779700 1 -- 172.21.15.103:0/982166462 >> v1:172.21.15.103:6789/0 conn(0x7f3c38132f50 legacy=0x7f3c3809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:03.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3d779700 1 -- 172.21.15.103:0/982166462 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c38132a20 con 0x7f3c38133360 2023-07-02T09:00:03.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3d779700 1 --2- 172.21.15.103:0/982166462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38133360 0x7f3c3809cc90 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c57496efbcfb7de server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:03.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3cf78700 1 -- 172.21.15.103:0/982166462 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3c28009cd0 con 0x7f3c38133360 2023-07-02T09:00:03.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3cf78700 1 -- 172.21.15.103:0/982166462 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3c28009e30 con 0x7f3c38133360 2023-07-02T09:00:03.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.673+0000 7f3c3cf78700 1 -- 172.21.15.103:0/982166462 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3c2800e360 con 0x7f3c38133360 2023-07-02T09:00:03.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.674+0000 7f3c401de700 1 -- 172.21.15.103:0/982166462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38133360 msgr2=0x7f3c3809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:03.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.674+0000 7f3c401de700 1 --2- 172.21.15.103:0/982166462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38133360 0x7f3c3809cc90 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:03.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.674+0000 7f3c401de700 1 -- 172.21.15.103:0/982166462 shutdown_connections 2023-07-02T09:00:03.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.674+0000 7f3c401de700 1 --2- 172.21.15.103:0/982166462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38133360 0x7f3c3809cc90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:03.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.674+0000 7f3c401de700 1 -- 172.21.15.103:0/982166462 >> 172.21.15.103:0/982166462 conn(0x7f3c380a2080 msgr2=0x7f3c380a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:03.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.674+0000 7f3c401de700 1 -- 172.21.15.103:0/982166462 shutdown_connections 2023-07-02T09:00:03.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.674+0000 7f3c401de700 1 -- 172.21.15.103:0/982166462 wait complete. 2023-07-02T09:00:03.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.674+0000 7f3c401de700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:03.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.675+0000 7f3c401de700 1 Processor -- start 2023-07-02T09:00:03.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.675+0000 7f3c401de700 1 -- start start 2023-07-02T09:00:03.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.676+0000 7f3c401de700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38132f50 0x7f3c3812bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:03.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.676+0000 7f3c401de700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3806d260 con 0x7f3c38132f50 2023-07-02T09:00:03.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.676+0000 7f3c3df7a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38132f50 0x7f3c3812bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:03.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.676+0000 7f3c3df7a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38132f50 0x7f3c3812bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55348/0 (socket says 172.21.15.103:55348) 2023-07-02T09:00:03.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.676+0000 7f3c3df7a700 1 -- 172.21.15.103:0/2556878325 learned_addr learned my addr 172.21.15.103:0/2556878325 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:03.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.676+0000 7f3c3df7a700 1 -- 172.21.15.103:0/2556878325 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3812bb90 con 0x7f3c38132f50 2023-07-02T09:00:03.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.677+0000 7f3c3df7a700 1 --2- 172.21.15.103:0/2556878325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38132f50 0x7f3c3812bea0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:03.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.677+0000 7f3c2e7fc700 1 -- 172.21.15.103:0/2556878325 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3c34005a60 con 0x7f3c38132f50 2023-07-02T09:00:03.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.677+0000 7f3c2e7fc700 1 -- 172.21.15.103:0/2556878325 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3c34005bc0 con 0x7f3c38132f50 2023-07-02T09:00:03.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.677+0000 7f3c2e7fc700 1 -- 172.21.15.103:0/2556878325 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3c34005e60 con 0x7f3c38132f50 2023-07-02T09:00:03.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.677+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3806f730 con 0x7f3c38132f50 2023-07-02T09:00:03.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.677+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3806f490 con 0x7f3c38132f50 2023-07-02T09:00:03.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.678+0000 7f3c2e7fc700 1 -- 172.21.15.103:0/2556878325 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3c340070b0 con 0x7f3c38132f50 2023-07-02T09:00:03.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.678+0000 7f3c2e7fc700 1 --2- 172.21.15.103:0/2556878325 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c1c0377c0 0x7f3c1c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:03.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.678+0000 7f3c2e7fc700 1 -- 172.21.15.103:0/2556878325 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3c340469f0 con 0x7f3c38132f50 2023-07-02T09:00:03.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.678+0000 7f3c3d779700 1 --2- 172.21.15.103:0/2556878325 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c1c0377c0 0x7f3c1c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:03.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.678+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3806f490 con 0x7f3c38132f50 2023-07-02T09:00:03.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.678+0000 7f3c3d779700 1 --2- 172.21.15.103:0/2556878325 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c1c0377c0 0x7f3c1c039cd0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:03.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.683+0000 7f3c2e7fc700 1 -- 172.21.15.103:0/2556878325 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3c3806f490 con 0x7f3c38132f50 2023-07-02T09:00:03.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.884+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3c3806f490 con 0x7f3c38132f50 2023-07-02T09:00:03.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.884+0000 7f3c2e7fc700 1 -- 172.21.15.103:0/2556878325 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3c3806f490 con 0x7f3c38132f50 2023-07-02T09:00:03.885 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:03.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.886+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c1c0377c0 msgr2=0x7f3c1c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:03.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.886+0000 7f3c401de700 1 --2- 172.21.15.103:0/2556878325 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c1c0377c0 0x7f3c1c039cd0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:03.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.886+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38132f50 msgr2=0x7f3c3812bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:03.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.886+0000 7f3c401de700 1 --2- 172.21.15.103:0/2556878325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38132f50 0x7f3c3812bea0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:03.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.887+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 shutdown_connections 2023-07-02T09:00:03.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.887+0000 7f3c401de700 1 --2- 172.21.15.103:0/2556878325 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c1c0377c0 0x7f3c1c039cd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:03.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.887+0000 7f3c401de700 1 --2- 172.21.15.103:0/2556878325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c38132f50 0x7f3c3812bea0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:03.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.887+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 >> 172.21.15.103:0/2556878325 conn(0x7f3c380a2080 msgr2=0x7f3c381bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:03.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.887+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 shutdown_connections 2023-07-02T09:00:03.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:03.887+0000 7f3c401de700 1 -- 172.21.15.103:0/2556878325 wait complete. 2023-07-02T09:00:03.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:04.901 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:05.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.041+0000 7f32e8c00700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:05.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.046+0000 7f32e8c00700 1 Processor -- start 2023-07-02T09:00:05.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.046+0000 7f32e8c00700 1 -- start start 2023-07-02T09:00:05.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e8c00700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f32e4132f50 0x7f32e4133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:05.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e8c00700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e409e260 con 0x7f32e409c580 2023-07-02T09:00:05.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e8c00700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e40a1640 con 0x7f32e4132f50 2023-07-02T09:00:05.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f32e4132f50 0x7f32e4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:05.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f32e4132f50 0x7f32e4133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55364/0 (socket says 172.21.15.103:55364) 2023-07-02T09:00:05.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e259c700 1 -- 172.21.15.103:0/4233729652 learned_addr learned my addr 172.21.15.103:0/4233729652 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:05.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e1d9b700 1 --1- 172.21.15.103:0/4233729652 >> v1:172.21.15.103:6789/0 conn(0x7f32e409c580 0x7f32e409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43202/0 (socket says 172.21.15.103:43202) 2023-07-02T09:00:05.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e259c700 1 -- 172.21.15.103:0/4233729652 >> v1:172.21.15.103:6789/0 conn(0x7f32e409c580 legacy=0x7f32e409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:05.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e259c700 1 -- 172.21.15.103:0/4233729652 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e4132a20 con 0x7f32e4132f50 2023-07-02T09:00:05.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e159a700 1 -- 172.21.15.103:0/4233729652 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278179245 0 0) 0x7f32e409e260 con 0x7f32e409c580 2023-07-02T09:00:05.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e259c700 1 --2- 172.21.15.103:0/4233729652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e4132f50 0x7f32e4133360 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb05eac776ddb761 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:05.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e159a700 1 -- 172.21.15.103:0/4233729652 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f32d4009d90 con 0x7f32e4132f50 2023-07-02T09:00:05.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e159a700 1 -- 172.21.15.103:0/4233729652 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f32d400e230 con 0x7f32e4132f50 2023-07-02T09:00:05.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.047+0000 7f32e159a700 1 -- 172.21.15.103:0/4233729652 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f32d400e4d0 con 0x7f32e4132f50 2023-07-02T09:00:05.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.048+0000 7f32e8c00700 1 -- 172.21.15.103:0/4233729652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e4132f50 msgr2=0x7f32e4133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:05.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.048+0000 7f32e8c00700 1 --2- 172.21.15.103:0/4233729652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e4132f50 0x7f32e4133360 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:05.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.048+0000 7f32e8c00700 1 -- 172.21.15.103:0/4233729652 shutdown_connections 2023-07-02T09:00:05.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.048+0000 7f32e8c00700 1 --2- 172.21.15.103:0/4233729652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e4132f50 0x7f32e4133360 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:05.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.048+0000 7f32e8c00700 1 -- 172.21.15.103:0/4233729652 >> 172.21.15.103:0/4233729652 conn(0x7f32e40a2080 msgr2=0x7f32e40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:05.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.048+0000 7f32e8c00700 1 -- 172.21.15.103:0/4233729652 shutdown_connections 2023-07-02T09:00:05.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.048+0000 7f32e8c00700 1 -- 172.21.15.103:0/4233729652 wait complete. 2023-07-02T09:00:05.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.049+0000 7f32e8c00700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:05.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.050+0000 7f32e8c00700 1 Processor -- start 2023-07-02T09:00:05.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.050+0000 7f32e8c00700 1 -- start start 2023-07-02T09:00:05.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.050+0000 7f32e8c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e409c580 0x7f32e41baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:05.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.050+0000 7f32e8c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e41ba610 con 0x7f32e409c580 2023-07-02T09:00:05.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.050+0000 7f32e259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e409c580 0x7f32e41baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:05.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.050+0000 7f32e259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e409c580 0x7f32e41baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55378/0 (socket says 172.21.15.103:55378) 2023-07-02T09:00:05.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.050+0000 7f32e259c700 1 -- 172.21.15.103:0/1543744479 learned_addr learned my addr 172.21.15.103:0/1543744479 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:05.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.051+0000 7f32e259c700 1 -- 172.21.15.103:0/1543744479 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e406dfc0 con 0x7f32e409c580 2023-07-02T09:00:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.051+0000 7f32e259c700 1 --2- 172.21.15.103:0/1543744479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e409c580 0x7f32e41baa60 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.051+0000 7f32d2ffd700 1 -- 172.21.15.103:0/1543744479 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f32d4014410 con 0x7f32e409c580 2023-07-02T09:00:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.051+0000 7f32d2ffd700 1 -- 172.21.15.103:0/1543744479 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f32d4014570 con 0x7f32e409c580 2023-07-02T09:00:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.051+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e41ba270 con 0x7f32e409c580 2023-07-02T09:00:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.051+0000 7f32d2ffd700 1 -- 172.21.15.103:0/1543744479 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f32d4014810 con 0x7f32e409c580 2023-07-02T09:00:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.051+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e406dba0 con 0x7f32e409c580 2023-07-02T09:00:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.052+0000 7f32d2ffd700 1 -- 172.21.15.103:0/1543744479 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f32d40149c0 con 0x7f32e409c580 2023-07-02T09:00:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.052+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e41ba460 con 0x7f32e409c580 2023-07-02T09:00:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.052+0000 7f32d2ffd700 1 --2- 172.21.15.103:0/1543744479 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32c40377c0 0x7f32c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.052+0000 7f32d2ffd700 1 -- 172.21.15.103:0/1543744479 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f32d40482f0 con 0x7f32e409c580 2023-07-02T09:00:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.052+0000 7f32e1d9b700 1 --2- 172.21.15.103:0/1543744479 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32c40377c0 0x7f32c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.053+0000 7f32e1d9b700 1 --2- 172.21.15.103:0/1543744479 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32c40377c0 0x7f32c4039cd0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:05.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.057+0000 7f32d2ffd700 1 -- 172.21.15.103:0/1543744479 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f32d400f8f0 con 0x7f32e409c580 2023-07-02T09:00:05.252 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.251+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f32e4061ea0 con 0x7f32e409c580 2023-07-02T09:00:05.252 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.252+0000 7f32d2ffd700 1 -- 172.21.15.103:0/1543744479 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f32d400f710 con 0x7f32e409c580 2023-07-02T09:00:05.253 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32c40377c0 msgr2=0x7f32c4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 --2- 172.21.15.103:0/1543744479 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32c40377c0 0x7f32c4039cd0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e409c580 msgr2=0x7f32e41baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 --2- 172.21.15.103:0/1543744479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e409c580 0x7f32e41baa60 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 shutdown_connections 2023-07-02T09:00:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 --2- 172.21.15.103:0/1543744479 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32c40377c0 0x7f32c4039cd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:05.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 --2- 172.21.15.103:0/1543744479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e409c580 0x7f32e41baa60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:05.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 >> 172.21.15.103:0/1543744479 conn(0x7f32e40a2080 msgr2=0x7f32e4130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:05.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 shutdown_connections 2023-07-02T09:00:05.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:05.254+0000 7f32e8c00700 1 -- 172.21.15.103:0/1543744479 wait complete. 2023-07-02T09:00:05.268 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:06.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:06.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.409+0000 7f3fe8d6a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:06.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.414+0000 7f3fe8d6a700 1 Processor -- start 2023-07-02T09:00:06.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.414+0000 7f3fe8d6a700 1 -- start start 2023-07-02T09:00:06.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.415+0000 7f3fe8d6a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3fe4132f50 0x7f3fe4133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:06.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.415+0000 7f3fe8d6a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe409e260 con 0x7f3fe409c580 2023-07-02T09:00:06.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.415+0000 7f3fe8d6a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe40a1640 con 0x7f3fe4132f50 2023-07-02T09:00:06.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.415+0000 7f3fe259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3fe4132f50 0x7f3fe4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:06.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.415+0000 7f3fe1d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3fe409c580 0x7f3fe409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43204/0 (socket says 172.21.15.103:43204) 2023-07-02T09:00:06.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.415+0000 7f3fe1d9b700 1 -- 172.21.15.103:0/2328676122 learned_addr learned my addr 172.21.15.103:0/2328676122 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:06.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.415+0000 7f3fe259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3fe4132f50 0x7f3fe4133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55386/0 (socket says 172.21.15.103:55386) 2023-07-02T09:00:06.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.416+0000 7f3fe259c700 1 -- 172.21.15.103:0/2328676122 >> v1:172.21.15.103:6789/0 conn(0x7f3fe409c580 legacy=0x7f3fe409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:06.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.416+0000 7f3fe259c700 1 -- 172.21.15.103:0/2328676122 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4132a20 con 0x7f3fe4132f50 2023-07-02T09:00:06.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.416+0000 7f3fe259c700 1 --2- 172.21.15.103:0/2328676122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe4132f50 0x7f3fe4133360 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5d2601b163d9a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:06.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.416+0000 7f3fe159a700 1 -- 172.21.15.103:0/2328676122 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3fd8009cd0 con 0x7f3fe4132f50 2023-07-02T09:00:06.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.416+0000 7f3fe159a700 1 -- 172.21.15.103:0/2328676122 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3fd8009e30 con 0x7f3fe4132f50 2023-07-02T09:00:06.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.416+0000 7f3fe159a700 1 -- 172.21.15.103:0/2328676122 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3fd800e360 con 0x7f3fe4132f50 2023-07-02T09:00:06.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.417+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/2328676122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe4132f50 msgr2=0x7f3fe4133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:06.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.417+0000 7f3fe8d6a700 1 --2- 172.21.15.103:0/2328676122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe4132f50 0x7f3fe4133360 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:06.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.417+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/2328676122 shutdown_connections 2023-07-02T09:00:06.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.417+0000 7f3fe8d6a700 1 --2- 172.21.15.103:0/2328676122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe4132f50 0x7f3fe4133360 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:06.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.417+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/2328676122 >> 172.21.15.103:0/2328676122 conn(0x7f3fe40a2080 msgr2=0x7f3fe40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:06.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.417+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/2328676122 shutdown_connections 2023-07-02T09:00:06.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.417+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/2328676122 wait complete. 2023-07-02T09:00:06.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.417+0000 7f3fe8d6a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:06.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.418+0000 7f3fe8d6a700 1 Processor -- start 2023-07-02T09:00:06.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.418+0000 7f3fe8d6a700 1 -- start start 2023-07-02T09:00:06.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe8d6a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe409c580 0x7f3fe412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:06.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe8d6a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe406f9d0 con 0x7f3fe409c580 2023-07-02T09:00:06.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe409c580 0x7f3fe412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:06.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe409c580 0x7f3fe412b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55394/0 (socket says 172.21.15.103:55394) 2023-07-02T09:00:06.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe259c700 1 -- 172.21.15.103:0/1604861406 learned_addr learned my addr 172.21.15.103:0/1604861406 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:06.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe259c700 1 -- 172.21.15.103:0/1604861406 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4133a00 con 0x7f3fe409c580 2023-07-02T09:00:06.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe259c700 1 --2- 172.21.15.103:0/1604861406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe409c580 0x7f3fe412b7b0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:06.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fd2ffd700 1 -- 172.21.15.103:0/1604861406 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3fd8013530 con 0x7f3fe409c580 2023-07-02T09:00:06.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fd2ffd700 1 -- 172.21.15.103:0/1604861406 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3fd8013690 con 0x7f3fe409c580 2023-07-02T09:00:06.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fd2ffd700 1 -- 172.21.15.103:0/1604861406 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3fd8013930 con 0x7f3fe409c580 2023-07-02T09:00:06.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe406d160 con 0x7f3fe409c580 2023-07-02T09:00:06.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.419+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe406da70 con 0x7f3fe409c580 2023-07-02T09:00:06.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.420+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4061ea0 con 0x7f3fe409c580 2023-07-02T09:00:06.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.420+0000 7f3fd2ffd700 1 -- 172.21.15.103:0/1604861406 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3fd8013ae0 con 0x7f3fe409c580 2023-07-02T09:00:06.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.420+0000 7f3fd2ffd700 1 --2- 172.21.15.103:0/1604861406 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fc40377c0 0x7f3fc4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:06.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.421+0000 7f3fd2ffd700 1 -- 172.21.15.103:0/1604861406 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3fd8018040 con 0x7f3fe409c580 2023-07-02T09:00:06.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.421+0000 7f3fe1d9b700 1 --2- 172.21.15.103:0/1604861406 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fc40377c0 0x7f3fc4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:06.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.421+0000 7f3fe1d9b700 1 --2- 172.21.15.103:0/1604861406 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fc40377c0 0x7f3fc4039cd0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:06.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.425+0000 7f3fd2ffd700 1 -- 172.21.15.103:0/1604861406 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3fd8018dd0 con 0x7f3fe409c580 2023-07-02T09:00:06.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.617+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3fe40725b0 con 0x7f3fe409c580 2023-07-02T09:00:06.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.617+0000 7f3fd2ffd700 1 -- 172.21.15.103:0/1604861406 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3fd80189f0 con 0x7f3fe409c580 2023-07-02T09:00:06.618 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:06.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fc40377c0 msgr2=0x7f3fc4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:06.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 --2- 172.21.15.103:0/1604861406 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fc40377c0 0x7f3fc4039cd0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:06.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe409c580 msgr2=0x7f3fe412b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 --2- 172.21.15.103:0/1604861406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe409c580 0x7f3fe412b7b0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 shutdown_connections 2023-07-02T09:00:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 --2- 172.21.15.103:0/1604861406 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fc40377c0 0x7f3fc4039cd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 --2- 172.21.15.103:0/1604861406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fe409c580 0x7f3fe412b7b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 >> 172.21.15.103:0/1604861406 conn(0x7f3fe40a2080 msgr2=0x7f3fe41babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 shutdown_connections 2023-07-02T09:00:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:06.620+0000 7f3fe8d6a700 1 -- 172.21.15.103:0/1604861406 wait complete. 2023-07-02T09:00:06.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:07.633 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:07.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.775+0000 7f2d00baf700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:07.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.780+0000 7f2d00baf700 1 Processor -- start 2023-07-02T09:00:07.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.780+0000 7f2d00baf700 1 -- start start 2023-07-02T09:00:07.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2d00baf700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2cfc133360 0x7f2cfc09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:07.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2d00baf700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc09e260 con 0x7f2cfc132f50 2023-07-02T09:00:07.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2d00baf700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0a1640 con 0x7f2cfc133360 2023-07-02T09:00:07.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2cf9d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2cfc133360 0x7f2cfc09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:07.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2cf9d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2cfc133360 0x7f2cfc09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55408/0 (socket says 172.21.15.103:55408) 2023-07-02T09:00:07.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2cfa59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2cfc132f50 0x7f2cfc09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43210/0 (socket says 172.21.15.103:43210) 2023-07-02T09:00:07.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2cf9d9b700 1 -- 172.21.15.103:0/4103301784 learned_addr learned my addr 172.21.15.103:0/4103301784 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:07.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2cf9d9b700 1 -- 172.21.15.103:0/4103301784 >> v1:172.21.15.103:6789/0 conn(0x7f2cfc132f50 legacy=0x7f2cfc09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2cf9d9b700 1 -- 172.21.15.103:0/4103301784 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc132a20 con 0x7f2cfc133360 2023-07-02T09:00:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.781+0000 7f2cf9d9b700 1 --2- 172.21.15.103:0/4103301784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc133360 0x7f2cfc09cc90 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d3b999472ea1eb server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:07.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2cf959a700 1 -- 172.21.15.103:0/4103301784 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2cec009d90 con 0x7f2cfc133360 2023-07-02T09:00:07.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2cf959a700 1 -- 172.21.15.103:0/4103301784 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2cec00e230 con 0x7f2cfc133360 2023-07-02T09:00:07.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2cf959a700 1 -- 172.21.15.103:0/4103301784 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2cec00e4d0 con 0x7f2cfc133360 2023-07-02T09:00:07.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2d00baf700 1 -- 172.21.15.103:0/4103301784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc133360 msgr2=0x7f2cfc09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:07.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2d00baf700 1 --2- 172.21.15.103:0/4103301784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc133360 0x7f2cfc09cc90 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:07.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2d00baf700 1 -- 172.21.15.103:0/4103301784 shutdown_connections 2023-07-02T09:00:07.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2d00baf700 1 --2- 172.21.15.103:0/4103301784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc133360 0x7f2cfc09cc90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:07.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2d00baf700 1 -- 172.21.15.103:0/4103301784 >> 172.21.15.103:0/4103301784 conn(0x7f2cfc0a2080 msgr2=0x7f2cfc0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:07.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2d00baf700 1 -- 172.21.15.103:0/4103301784 shutdown_connections 2023-07-02T09:00:07.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.782+0000 7f2d00baf700 1 -- 172.21.15.103:0/4103301784 wait complete. 2023-07-02T09:00:07.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.783+0000 7f2d00baf700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:07.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.784+0000 7f2d00baf700 1 Processor -- start 2023-07-02T09:00:07.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.784+0000 7f2d00baf700 1 -- start start 2023-07-02T09:00:07.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.784+0000 7f2d00baf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc132f50 0x7f2cfc12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:07.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.784+0000 7f2d00baf700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc06d260 con 0x7f2cfc132f50 2023-07-02T09:00:07.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2cfa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc132f50 0x7f2cfc12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:07.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2cfa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc132f50 0x7f2cfc12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55418/0 (socket says 172.21.15.103:55418) 2023-07-02T09:00:07.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2cfa59c700 1 -- 172.21.15.103:0/1112250631 learned_addr learned my addr 172.21.15.103:0/1112250631 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:07.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2cfa59c700 1 -- 172.21.15.103:0/1112250631 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc12bb90 con 0x7f2cfc132f50 2023-07-02T09:00:07.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2cfa59c700 1 --2- 172.21.15.103:0/1112250631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc132f50 0x7f2cfc12bea0 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:07.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2ceaffd700 1 -- 172.21.15.103:0/1112250631 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2cf00059a0 con 0x7f2cfc132f50 2023-07-02T09:00:07.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2ceaffd700 1 -- 172.21.15.103:0/1112250631 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2cf0005b00 con 0x7f2cfc132f50 2023-07-02T09:00:07.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc06f730 con 0x7f2cfc132f50 2023-07-02T09:00:07.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2ceaffd700 1 -- 172.21.15.103:0/1112250631 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2cf0005da0 con 0x7f2cfc132f50 2023-07-02T09:00:07.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.785+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc06f490 con 0x7f2cfc132f50 2023-07-02T09:00:07.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.786+0000 7f2ceaffd700 1 -- 172.21.15.103:0/1112250631 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2cf00070b0 con 0x7f2cfc132f50 2023-07-02T09:00:07.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.786+0000 7f2ceaffd700 1 --2- 172.21.15.103:0/1112250631 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2cdc0377c0 0x7f2cdc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:07.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.786+0000 7f2ceaffd700 1 -- 172.21.15.103:0/1112250631 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2cf00469f0 con 0x7f2cfc132f50 2023-07-02T09:00:07.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.786+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc06f490 con 0x7f2cfc132f50 2023-07-02T09:00:07.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.786+0000 7f2cf9d9b700 1 --2- 172.21.15.103:0/1112250631 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2cdc0377c0 0x7f2cdc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:07.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.787+0000 7f2cf9d9b700 1 --2- 172.21.15.103:0/1112250631 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2cdc0377c0 0x7f2cdc039cd0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:07.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.791+0000 7f2ceaffd700 1 -- 172.21.15.103:0/1112250631 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2cfc06f490 con 0x7f2cfc132f50 2023-07-02T09:00:07.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2023-07-02T09:00:07.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.984+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2cfc06f490 con 0x7f2cfc132f50 2023-07-02T09:00:07.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.984+0000 7f2ceaffd700 1 -- 172.21.15.103:0/1112250631 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2cfc06f490 con 0x7f2cfc132f50 2023-07-02T09:00:07.985 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:07.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2cdc0377c0 msgr2=0x7f2cdc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:07.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 --2- 172.21.15.103:0/1112250631 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2cdc0377c0 0x7f2cdc039cd0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:07.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc132f50 msgr2=0x7f2cfc12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:07.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 --2- 172.21.15.103:0/1112250631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc132f50 0x7f2cfc12bea0 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:07.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 shutdown_connections 2023-07-02T09:00:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 --2- 172.21.15.103:0/1112250631 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2cdc0377c0 0x7f2cdc039cd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 --2- 172.21.15.103:0/1112250631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cfc132f50 0x7f2cfc12bea0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 >> 172.21.15.103:0/1112250631 conn(0x7f2cfc0a2080 msgr2=0x7f2cfc1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 shutdown_connections 2023-07-02T09:00:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:07.986+0000 7f2d00baf700 1 -- 172.21.15.103:0/1112250631 wait complete. 2023-07-02T09:00:07.999 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:09.000 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.139+0000 7f421e8f0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.144+0000 7f421e8f0700 1 Processor -- start 2023-07-02T09:00:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.144+0000 7f421e8f0700 1 -- start start 2023-07-02T09:00:09.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.145+0000 7f421e8f0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f421812f250 0x7f421812f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:09.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.145+0000 7f421e8f0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421806df60 con 0x7f4218098a00 2023-07-02T09:00:09.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.145+0000 7f421e8f0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421809a710 con 0x7f421812f250 2023-07-02T09:00:09.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.145+0000 7f4217fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f421812f250 0x7f421812f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:09.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.145+0000 7f42177fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4218098a00 0x7f4218098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43224/0 (socket says 172.21.15.103:43224) 2023-07-02T09:00:09.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.145+0000 7f42177fe700 1 -- 172.21.15.103:0/2019252538 learned_addr learned my addr 172.21.15.103:0/2019252538 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:09.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.145+0000 7f4217fff700 1 -- 172.21.15.103:0/2019252538 >> v1:172.21.15.103:6789/0 conn(0x7f4218098a00 legacy=0x7f4218098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:09.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.145+0000 7f4217fff700 1 -- 172.21.15.103:0/2019252538 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421812ed20 con 0x7f421812f250 2023-07-02T09:00:09.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.146+0000 7f4217fff700 1 --2- 172.21.15.103:0/2019252538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421812f250 0x7f421812f660 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cf08be2d541097 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:09.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.146+0000 7f4216ffd700 1 -- 172.21.15.103:0/2019252538 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4200009cd0 con 0x7f421812f250 2023-07-02T09:00:09.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.146+0000 7f4216ffd700 1 -- 172.21.15.103:0/2019252538 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4200009e30 con 0x7f421812f250 2023-07-02T09:00:09.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.146+0000 7f4216ffd700 1 -- 172.21.15.103:0/2019252538 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f420000f470 con 0x7f421812f250 2023-07-02T09:00:09.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.147+0000 7f421e8f0700 1 -- 172.21.15.103:0/2019252538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421812f250 msgr2=0x7f421812f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:09.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.147+0000 7f421e8f0700 1 --2- 172.21.15.103:0/2019252538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421812f250 0x7f421812f660 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:09.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.147+0000 7f421e8f0700 1 -- 172.21.15.103:0/2019252538 shutdown_connections 2023-07-02T09:00:09.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.147+0000 7f421e8f0700 1 --2- 172.21.15.103:0/2019252538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421812f250 0x7f421812f660 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:09.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.147+0000 7f421e8f0700 1 -- 172.21.15.103:0/2019252538 >> 172.21.15.103:0/2019252538 conn(0x7f421809d810 msgr2=0x7f421809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:09.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.147+0000 7f421e8f0700 1 -- 172.21.15.103:0/2019252538 shutdown_connections 2023-07-02T09:00:09.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.147+0000 7f421e8f0700 1 -- 172.21.15.103:0/2019252538 wait complete. 2023-07-02T09:00:09.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.147+0000 7f421e8f0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:09.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.148+0000 7f421e8f0700 1 Processor -- start 2023-07-02T09:00:09.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.148+0000 7f421e8f0700 1 -- start start 2023-07-02T09:00:09.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f421e8f0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4218098a00 0x7f42181333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:09.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f421e8f0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421812c040 con 0x7f4218098a00 2023-07-02T09:00:09.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f4217fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4218098a00 0x7f42181333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:09.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f4217fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4218098a00 0x7f42181333d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55432/0 (socket says 172.21.15.103:55432) 2023-07-02T09:00:09.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f4217fff700 1 -- 172.21.15.103:0/4273823311 learned_addr learned my addr 172.21.15.103:0/4273823311 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:09.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f4217fff700 1 -- 172.21.15.103:0/4273823311 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4218099b60 con 0x7f4218098a00 2023-07-02T09:00:09.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f4217fff700 1 --2- 172.21.15.103:0/4273823311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4218098a00 0x7f42181333d0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:09.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f4214ff9700 1 -- 172.21.15.103:0/4273823311 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4200012e90 con 0x7f4218098a00 2023-07-02T09:00:09.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f4214ff9700 1 -- 172.21.15.103:0/4273823311 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f42000103b0 con 0x7f4218098a00 2023-07-02T09:00:09.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f4214ff9700 1 -- 172.21.15.103:0/4273823311 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4200010650 con 0x7f4218098a00 2023-07-02T09:00:09.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42181350c0 con 0x7f4218098a00 2023-07-02T09:00:09.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.149+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421812c520 con 0x7f4218098a00 2023-07-02T09:00:09.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.150+0000 7f4214ff9700 1 -- 172.21.15.103:0/4273823311 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4200010800 con 0x7f4218098a00 2023-07-02T09:00:09.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.150+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4218061ea0 con 0x7f4218098a00 2023-07-02T09:00:09.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.150+0000 7f4214ff9700 1 --2- 172.21.15.103:0/4273823311 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f41f80377c0 0x7f41f8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:09.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.150+0000 7f4214ff9700 1 -- 172.21.15.103:0/4273823311 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4200047970 con 0x7f4218098a00 2023-07-02T09:00:09.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.151+0000 7f42177fe700 1 --2- 172.21.15.103:0/4273823311 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f41f80377c0 0x7f41f8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:09.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.151+0000 7f42177fe700 1 --2- 172.21.15.103:0/4273823311 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f41f80377c0 0x7f41f8039cd0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:09.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.155+0000 7f4214ff9700 1 -- 172.21.15.103:0/4273823311 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f42000165a0 con 0x7f4218098a00 2023-07-02T09:00:09.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.348+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4218099b60 con 0x7f4218098a00 2023-07-02T09:00:09.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.348+0000 7f4214ff9700 1 -- 172.21.15.103:0/4273823311 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4218099b60 con 0x7f4218098a00 2023-07-02T09:00:09.349 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:09.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f41f80377c0 msgr2=0x7f41f8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:09.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 --2- 172.21.15.103:0/4273823311 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f41f80377c0 0x7f41f8039cd0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:09.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4218098a00 msgr2=0x7f42181333d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:09.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 --2- 172.21.15.103:0/4273823311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4218098a00 0x7f42181333d0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:09.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 shutdown_connections 2023-07-02T09:00:09.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 --2- 172.21.15.103:0/4273823311 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f41f80377c0 0x7f41f8039cd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:09.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 --2- 172.21.15.103:0/4273823311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4218098a00 0x7f42181333d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:09.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 >> 172.21.15.103:0/4273823311 conn(0x7f421809d810 msgr2=0x7f421812cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:09.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 shutdown_connections 2023-07-02T09:00:09.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:09.350+0000 7f421e8f0700 1 -- 172.21.15.103:0/4273823311 wait complete. 2023-07-02T09:00:09.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:10.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:10.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.503+0000 7f5c8b7b3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:10.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.508+0000 7f5c8b7b3700 1 Processor -- start 2023-07-02T09:00:10.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.508+0000 7f5c8b7b3700 1 -- start start 2023-07-02T09:00:10.510 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c8b7b3700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5c84132f50 0x7f5c84133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:10.510 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c8b7b3700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8409e260 con 0x7f5c8409c580 2023-07-02T09:00:10.510 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c8b7b3700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c840a1640 con 0x7f5c84132f50 2023-07-02T09:00:10.510 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c8954f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5c84132f50 0x7f5c84133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:10.510 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c88d4e700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f5c8409c580 0x7f5c8409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43228/0 (socket says 172.21.15.103:43228) 2023-07-02T09:00:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c88d4e700 1 -- 172.21.15.103:0/2963592228 learned_addr learned my addr 172.21.15.103:0/2963592228 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c7bfff700 1 -- 172.21.15.103:0/2963592228 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796701185 0 0) 0x7f5c8409e260 con 0x7f5c8409c580 2023-07-02T09:00:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c7bfff700 1 -- 172.21.15.103:0/2963592228 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c70003580 con 0x7f5c8409c580 2023-07-02T09:00:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c8954f700 1 -- 172.21.15.103:0/2963592228 >> v1:172.21.15.103:6789/0 conn(0x7f5c8409c580 legacy=0x7f5c8409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c8954f700 1 -- 172.21.15.103:0/2963592228 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c84132a20 con 0x7f5c84132f50 2023-07-02T09:00:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c8954f700 1 --2- 172.21.15.103:0/2963592228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c84132f50 0x7f5c84133360 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579cfc19a8638dfc server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c7bfff700 1 -- 172.21.15.103:0/2963592228 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5c80009c80 con 0x7f5c84132f50 2023-07-02T09:00:10.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c7bfff700 1 -- 172.21.15.103:0/2963592228 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5c80009de0 con 0x7f5c84132f50 2023-07-02T09:00:10.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.509+0000 7f5c7bfff700 1 -- 172.21.15.103:0/2963592228 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5c8000e360 con 0x7f5c84132f50 2023-07-02T09:00:10.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.510+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2963592228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c84132f50 msgr2=0x7f5c84133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:10.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.510+0000 7f5c8b7b3700 1 --2- 172.21.15.103:0/2963592228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c84132f50 0x7f5c84133360 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:10.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.510+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2963592228 shutdown_connections 2023-07-02T09:00:10.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.510+0000 7f5c8b7b3700 1 --2- 172.21.15.103:0/2963592228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c84132f50 0x7f5c84133360 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:10.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.510+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2963592228 >> 172.21.15.103:0/2963592228 conn(0x7f5c840a2080 msgr2=0x7f5c840a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:10.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.510+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2963592228 shutdown_connections 2023-07-02T09:00:10.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.510+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2963592228 wait complete. 2023-07-02T09:00:10.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.510+0000 7f5c8b7b3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:10.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.511+0000 7f5c8b7b3700 1 Processor -- start 2023-07-02T09:00:10.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.511+0000 7f5c8b7b3700 1 -- start start 2023-07-02T09:00:10.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.512+0000 7f5c8b7b3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c8409c580 0x7f5c841baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:10.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.512+0000 7f5c8b7b3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c841ba610 con 0x7f5c8409c580 2023-07-02T09:00:10.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.512+0000 7f5c8954f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c8409c580 0x7f5c841baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:10.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.512+0000 7f5c8954f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c8409c580 0x7f5c841baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55452/0 (socket says 172.21.15.103:55452) 2023-07-02T09:00:10.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.512+0000 7f5c8954f700 1 -- 172.21.15.103:0/2202932556 learned_addr learned my addr 172.21.15.103:0/2202932556 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:10.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.512+0000 7f5c8954f700 1 -- 172.21.15.103:0/2202932556 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8406dfc0 con 0x7f5c8409c580 2023-07-02T09:00:10.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.512+0000 7f5c8954f700 1 --2- 172.21.15.103:0/2202932556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c8409c580 0x7f5c841baa60 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:10.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.513+0000 7f5c79ffb700 1 -- 172.21.15.103:0/2202932556 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5c80013530 con 0x7f5c8409c580 2023-07-02T09:00:10.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.513+0000 7f5c79ffb700 1 -- 172.21.15.103:0/2202932556 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5c80013690 con 0x7f5c8409c580 2023-07-02T09:00:10.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.513+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c841ba270 con 0x7f5c8409c580 2023-07-02T09:00:10.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.513+0000 7f5c79ffb700 1 -- 172.21.15.103:0/2202932556 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5c80013930 con 0x7f5c8409c580 2023-07-02T09:00:10.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.513+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8406dba0 con 0x7f5c8409c580 2023-07-02T09:00:10.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.514+0000 7f5c79ffb700 1 -- 172.21.15.103:0/2202932556 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f5c80013ae0 con 0x7f5c8409c580 2023-07-02T09:00:10.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.514+0000 7f5c79ffb700 1 --2- 172.21.15.103:0/2202932556 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5c680377c0 0x7f5c68039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:10.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.514+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c841ba460 con 0x7f5c8409c580 2023-07-02T09:00:10.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.514+0000 7f5c79ffb700 1 -- 172.21.15.103:0/2202932556 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f5c80018040 con 0x7f5c8409c580 2023-07-02T09:00:10.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.514+0000 7f5c88d4e700 1 --2- 172.21.15.103:0/2202932556 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5c680377c0 0x7f5c68039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:10.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.515+0000 7f5c88d4e700 1 --2- 172.21.15.103:0/2202932556 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5c680377c0 0x7f5c68039cd0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:10.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.519+0000 7f5c79ffb700 1 -- 172.21.15.103:0/2202932556 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f5c80018850 con 0x7f5c8409c580 2023-07-02T09:00:10.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.712+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5c84061ea0 con 0x7f5c8409c580 2023-07-02T09:00:10.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.712+0000 7f5c79ffb700 1 -- 172.21.15.103:0/2202932556 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f5c80010d20 con 0x7f5c8409c580 2023-07-02T09:00:10.712 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:10.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5c680377c0 msgr2=0x7f5c68039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:10.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 --2- 172.21.15.103:0/2202932556 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5c680377c0 0x7f5c68039cd0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:10.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c8409c580 msgr2=0x7f5c841baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:10.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 --2- 172.21.15.103:0/2202932556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c8409c580 0x7f5c841baa60 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:10.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 shutdown_connections 2023-07-02T09:00:10.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 --2- 172.21.15.103:0/2202932556 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5c680377c0 0x7f5c68039cd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:10.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 --2- 172.21.15.103:0/2202932556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c8409c580 0x7f5c841baa60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:10.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 >> 172.21.15.103:0/2202932556 conn(0x7f5c840a2080 msgr2=0x7f5c84130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:10.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 shutdown_connections 2023-07-02T09:00:10.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:10.714+0000 7f5c8b7b3700 1 -- 172.21.15.103:0/2202932556 wait complete. 2023-07-02T09:00:10.727 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:11.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:11.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.870+0000 7fd713265700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:11.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.875+0000 7fd713265700 1 Processor -- start 2023-07-02T09:00:11.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.875+0000 7fd713265700 1 -- start start 2023-07-02T09:00:11.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd713265700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd70c132f50 0x7fd70c133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:11.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd713265700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c09e260 con 0x7fd70c09c580 2023-07-02T09:00:11.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd713265700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70c0a1640 con 0x7fd70c132f50 2023-07-02T09:00:11.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd710800700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd70c09c580 0x7fd70c09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43244/0 (socket says 172.21.15.103:43244) 2023-07-02T09:00:11.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd710800700 1 -- 172.21.15.103:0/242158860 learned_addr learned my addr 172.21.15.103:0/242158860 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:11.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd711001700 1 --2- 172.21.15.103:0/242158860 >> v2:172.21.15.103:3300/0 conn(0x7fd70c132f50 0x7fd70c133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:11.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd703fff700 1 -- 172.21.15.103:0/242158860 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557766077 0 0) 0x7fd70c09e260 con 0x7fd70c09c580 2023-07-02T09:00:11.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd703fff700 1 -- 172.21.15.103:0/242158860 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f8003580 con 0x7fd70c09c580 2023-07-02T09:00:11.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd711001700 1 -- 172.21.15.103:0/242158860 >> v1:172.21.15.103:6789/0 conn(0x7fd70c09c580 legacy=0x7fd70c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:11.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.876+0000 7fd711001700 1 -- 172.21.15.103:0/242158860 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c132a20 con 0x7fd70c132f50 2023-07-02T09:00:11.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.877+0000 7fd711001700 1 --2- 172.21.15.103:0/242158860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c132f50 0x7fd70c133360 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3e01a37ac6432 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:11.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.877+0000 7fd703fff700 1 -- 172.21.15.103:0/242158860 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd708009c80 con 0x7fd70c132f50 2023-07-02T09:00:11.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.877+0000 7fd703fff700 1 -- 172.21.15.103:0/242158860 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd708009de0 con 0x7fd70c132f50 2023-07-02T09:00:11.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.877+0000 7fd703fff700 1 -- 172.21.15.103:0/242158860 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd70800e360 con 0x7fd70c132f50 2023-07-02T09:00:11.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.877+0000 7fd713265700 1 -- 172.21.15.103:0/242158860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c132f50 msgr2=0x7fd70c133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:11.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.877+0000 7fd713265700 1 --2- 172.21.15.103:0/242158860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c132f50 0x7fd70c133360 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:11.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.877+0000 7fd713265700 1 -- 172.21.15.103:0/242158860 shutdown_connections 2023-07-02T09:00:11.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.877+0000 7fd713265700 1 --2- 172.21.15.103:0/242158860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c132f50 0x7fd70c133360 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:11.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.878+0000 7fd713265700 1 -- 172.21.15.103:0/242158860 >> 172.21.15.103:0/242158860 conn(0x7fd70c0a2080 msgr2=0x7fd70c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:11.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.878+0000 7fd713265700 1 -- 172.21.15.103:0/242158860 shutdown_connections 2023-07-02T09:00:11.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.878+0000 7fd713265700 1 -- 172.21.15.103:0/242158860 wait complete. 2023-07-02T09:00:11.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.878+0000 7fd713265700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:11.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.879+0000 7fd713265700 1 Processor -- start 2023-07-02T09:00:11.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.879+0000 7fd713265700 1 -- start start 2023-07-02T09:00:11.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.879+0000 7fd713265700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c09c580 0x7fd70c1baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:11.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.879+0000 7fd713265700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c1ba610 con 0x7fd70c09c580 2023-07-02T09:00:11.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.879+0000 7fd711001700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c09c580 0x7fd70c1baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:11.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd711001700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c09c580 0x7fd70c1baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55472/0 (socket says 172.21.15.103:55472) 2023-07-02T09:00:11.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd711001700 1 -- 172.21.15.103:0/3284378404 learned_addr learned my addr 172.21.15.103:0/3284378404 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:11.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd711001700 1 -- 172.21.15.103:0/3284378404 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c06dfc0 con 0x7fd70c09c580 2023-07-02T09:00:11.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd711001700 1 --2- 172.21.15.103:0/3284378404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c09c580 0x7fd70c1baa60 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:11.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd701ffb700 1 -- 172.21.15.103:0/3284378404 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd708014540 con 0x7fd70c09c580 2023-07-02T09:00:11.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd701ffb700 1 -- 172.21.15.103:0/3284378404 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd7080146a0 con 0x7fd70c09c580 2023-07-02T09:00:11.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd701ffb700 1 -- 172.21.15.103:0/3284378404 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd708014940 con 0x7fd70c09c580 2023-07-02T09:00:11.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c1ba270 con 0x7fd70c09c580 2023-07-02T09:00:11.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.880+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70c06dba0 con 0x7fd70c09c580 2023-07-02T09:00:11.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.881+0000 7fd701ffb700 1 -- 172.21.15.103:0/3284378404 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd708014aa0 con 0x7fd70c09c580 2023-07-02T09:00:11.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.881+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70c1ba460 con 0x7fd70c09c580 2023-07-02T09:00:11.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.881+0000 7fd701ffb700 1 --2- 172.21.15.103:0/3284378404 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6f00377c0 0x7fd6f0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:11.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.881+0000 7fd701ffb700 1 -- 172.21.15.103:0/3284378404 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd7080481b0 con 0x7fd70c09c580 2023-07-02T09:00:11.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.882+0000 7fd710800700 1 --2- 172.21.15.103:0/3284378404 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6f00377c0 0x7fd6f0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:11.886 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.882+0000 7fd710800700 1 --2- 172.21.15.103:0/3284378404 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6f00377c0 0x7fd6f0039cd0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:11.886 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:11.886+0000 7fd701ffb700 1 -- 172.21.15.103:0/3284378404 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd7080138c0 con 0x7fd70c09c580 2023-07-02T09:00:12.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.078+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd70c061ea0 con 0x7fd70c09c580 2023-07-02T09:00:12.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.079+0000 7fd701ffb700 1 -- 172.21.15.103:0/3284378404 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd70800f710 con 0x7fd70c09c580 2023-07-02T09:00:12.079 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:12.081 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6f00377c0 msgr2=0x7fd6f0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:12.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 --2- 172.21.15.103:0/3284378404 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6f00377c0 0x7fd6f0039cd0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:12.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c09c580 msgr2=0x7fd70c1baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:12.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 --2- 172.21.15.103:0/3284378404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c09c580 0x7fd70c1baa60 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:12.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 shutdown_connections 2023-07-02T09:00:12.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 --2- 172.21.15.103:0/3284378404 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6f00377c0 0x7fd6f0039cd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:12.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 --2- 172.21.15.103:0/3284378404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd70c09c580 0x7fd70c1baa60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:12.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 >> 172.21.15.103:0/3284378404 conn(0x7fd70c0a2080 msgr2=0x7fd70c130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:12.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 shutdown_connections 2023-07-02T09:00:12.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:12.081+0000 7fd713265700 1 -- 172.21.15.103:0/3284378404 wait complete. 2023-07-02T09:00:12.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:13.096 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:13.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.237+0000 7fdc5943e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:13.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.242+0000 7fdc5943e700 1 Processor -- start 2023-07-02T09:00:13.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.242+0000 7fdc5943e700 1 -- start start 2023-07-02T09:00:13.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc5943e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdc5409c720 0x7fdc5412f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:13.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc5943e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5406df60 con 0x7fdc5409cb30 2023-07-02T09:00:13.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc5943e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5409a710 con 0x7fdc5409c720 2023-07-02T09:00:13.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc52ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdc5409c720 0x7fdc5412f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:13.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc527fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdc5409cb30 0x7fdc54098a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43252/0 (socket says 172.21.15.103:43252) 2023-07-02T09:00:13.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc527fc700 1 -- 172.21.15.103:0/1094577661 learned_addr learned my addr 172.21.15.103:0/1094577661 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:13.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc52ffd700 1 -- 172.21.15.103:0/1094577661 >> v1:172.21.15.103:6789/0 conn(0x7fdc5409cb30 legacy=0x7fdc54098a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:13.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc52ffd700 1 -- 172.21.15.103:0/1094577661 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5412ed20 con 0x7fdc5409c720 2023-07-02T09:00:13.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc52ffd700 1 --2- 172.21.15.103:0/1094577661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc5412f250 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15f92896da01f80 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:13.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc51ffb700 1 -- 172.21.15.103:0/1094577661 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc48009cd0 con 0x7fdc5409c720 2023-07-02T09:00:13.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc51ffb700 1 -- 172.21.15.103:0/1094577661 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdc48009e30 con 0x7fdc5409c720 2023-07-02T09:00:13.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.243+0000 7fdc51ffb700 1 -- 172.21.15.103:0/1094577661 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc4800e360 con 0x7fdc5409c720 2023-07-02T09:00:13.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.244+0000 7fdc5943e700 1 -- 172.21.15.103:0/1094577661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 msgr2=0x7fdc5412f250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:13.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.244+0000 7fdc5943e700 1 --2- 172.21.15.103:0/1094577661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc5412f250 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:13.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.244+0000 7fdc5943e700 1 -- 172.21.15.103:0/1094577661 shutdown_connections 2023-07-02T09:00:13.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.244+0000 7fdc5943e700 1 --2- 172.21.15.103:0/1094577661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc5412f250 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:13.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.244+0000 7fdc5943e700 1 -- 172.21.15.103:0/1094577661 >> 172.21.15.103:0/1094577661 conn(0x7fdc5409d810 msgr2=0x7fdc5409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:13.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.244+0000 7fdc5943e700 1 -- 172.21.15.103:0/1094577661 shutdown_connections 2023-07-02T09:00:13.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.244+0000 7fdc5943e700 1 -- 172.21.15.103:0/1094577661 wait complete. 2023-07-02T09:00:13.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.244+0000 7fdc5943e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:13.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.245+0000 7fdc5943e700 1 Processor -- start 2023-07-02T09:00:13.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.245+0000 7fdc5943e700 1 -- start start 2023-07-02T09:00:13.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.246+0000 7fdc5943e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc54117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:13.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.246+0000 7fdc5943e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5411a5a0 con 0x7fdc5409c720 2023-07-02T09:00:13.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.246+0000 7fdc52ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc54117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:13.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.246+0000 7fdc52ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc54117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55500/0 (socket says 172.21.15.103:55500) 2023-07-02T09:00:13.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.246+0000 7fdc52ffd700 1 -- 172.21.15.103:0/1113490468 learned_addr learned my addr 172.21.15.103:0/1113490468 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:13.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.247+0000 7fdc52ffd700 1 -- 172.21.15.103:0/1113490468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5411a820 con 0x7fdc5409c720 2023-07-02T09:00:13.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.247+0000 7fdc52ffd700 1 --2- 172.21.15.103:0/1113490468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc54117360 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:13.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.247+0000 7fdc3b7fe700 1 -- 172.21.15.103:0/1113490468 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc48013530 con 0x7fdc5409c720 2023-07-02T09:00:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.247+0000 7fdc3b7fe700 1 -- 172.21.15.103:0/1113490468 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdc48013690 con 0x7fdc5409c720 2023-07-02T09:00:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.247+0000 7fdc3b7fe700 1 -- 172.21.15.103:0/1113490468 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc48013930 con 0x7fdc5409c720 2023-07-02T09:00:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.247+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc54099af0 con 0x7fdc5409c720 2023-07-02T09:00:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.247+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5411b460 con 0x7fdc5409c720 2023-07-02T09:00:13.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.248+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc54119000 con 0x7fdc5409c720 2023-07-02T09:00:13.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.248+0000 7fdc3b7fe700 1 -- 172.21.15.103:0/1113490468 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fdc48013ae0 con 0x7fdc5409c720 2023-07-02T09:00:13.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.248+0000 7fdc3b7fe700 1 --2- 172.21.15.103:0/1113490468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc400377c0 0x7fdc40039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:13.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.248+0000 7fdc3b7fe700 1 -- 172.21.15.103:0/1113490468 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fdc48018040 con 0x7fdc5409c720 2023-07-02T09:00:13.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.248+0000 7fdc527fc700 1 --2- 172.21.15.103:0/1113490468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc400377c0 0x7fdc40039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:13.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.249+0000 7fdc527fc700 1 --2- 172.21.15.103:0/1113490468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc400377c0 0x7fdc40039cd0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.253+0000 7fdc3b7fe700 1 -- 172.21.15.103:0/1113490468 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fdc48018dd0 con 0x7fdc5409c720 2023-07-02T09:00:13.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~28s 2023-07-02T09:00:13.446 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.446+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdc54061ea0 con 0x7fdc5409c720 2023-07-02T09:00:13.447 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.446+0000 7fdc3b7fe700 1 -- 172.21.15.103:0/1113490468 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fdc480189f0 con 0x7fdc5409c720 2023-07-02T09:00:13.447 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:13.449 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc400377c0 msgr2=0x7fdc40039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:13.449 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 --2- 172.21.15.103:0/1113490468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc400377c0 0x7fdc40039cd0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:13.449 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 msgr2=0x7fdc54117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 --2- 172.21.15.103:0/1113490468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc54117360 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 shutdown_connections 2023-07-02T09:00:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 --2- 172.21.15.103:0/1113490468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc400377c0 0x7fdc40039cd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 --2- 172.21.15.103:0/1113490468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc5409c720 0x7fdc54117360 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 >> 172.21.15.103:0/1113490468 conn(0x7fdc5409d810 msgr2=0x7fdc5409b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 shutdown_connections 2023-07-02T09:00:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:13.449+0000 7fdc5943e700 1 -- 172.21.15.103:0/1113490468 wait complete. 2023-07-02T09:00:13.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:14.463 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:14.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.601+0000 7ff2bc2d6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:14.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.606+0000 7ff2bc2d6700 1 Processor -- start 2023-07-02T09:00:14.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.606+0000 7ff2bc2d6700 1 -- start start 2023-07-02T09:00:14.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.607+0000 7ff2bc2d6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2b4133360 0x7ff2b409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.607+0000 7ff2bc2d6700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b409e260 con 0x7ff2b4132f50 2023-07-02T09:00:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.607+0000 7ff2bc2d6700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b40a1640 con 0x7ff2b4133360 2023-07-02T09:00:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.607+0000 7ff2b9871700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2b4133360 0x7ff2b409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.607+0000 7ff2ba072700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff2b4132f50 0x7ff2b409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52216/0 (socket says 172.21.15.103:52216) 2023-07-02T09:00:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.607+0000 7ff2ba072700 1 -- 172.21.15.103:0/3572136115 learned_addr learned my addr 172.21.15.103:0/3572136115 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:14.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.608+0000 7ff2b9871700 1 -- 172.21.15.103:0/3572136115 >> v1:172.21.15.103:6789/0 conn(0x7ff2b4132f50 legacy=0x7ff2b409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:14.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.608+0000 7ff2b9871700 1 -- 172.21.15.103:0/3572136115 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b4132a20 con 0x7ff2b4133360 2023-07-02T09:00:14.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.608+0000 7ff2b9070700 1 -- 172.21.15.103:0/3572136115 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757132748 0 0) 0x7ff2b409e260 con 0x7ff2b4132f50 2023-07-02T09:00:14.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.608+0000 7ff2b9871700 1 --2- 172.21.15.103:0/3572136115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4133360 0x7ff2b409cc90 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d0ad573507a5f7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:14.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.608+0000 7ff2b9070700 1 -- 172.21.15.103:0/3572136115 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff2b0009cd0 con 0x7ff2b4133360 2023-07-02T09:00:14.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.608+0000 7ff2b9070700 1 -- 172.21.15.103:0/3572136115 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff2b0009e30 con 0x7ff2b4133360 2023-07-02T09:00:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.608+0000 7ff2b9070700 1 -- 172.21.15.103:0/3572136115 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff2b000e360 con 0x7ff2b4133360 2023-07-02T09:00:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.609+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/3572136115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4133360 msgr2=0x7ff2b409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.609+0000 7ff2bc2d6700 1 --2- 172.21.15.103:0/3572136115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4133360 0x7ff2b409cc90 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.609+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/3572136115 shutdown_connections 2023-07-02T09:00:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.609+0000 7ff2bc2d6700 1 --2- 172.21.15.103:0/3572136115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4133360 0x7ff2b409cc90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.609+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/3572136115 >> 172.21.15.103:0/3572136115 conn(0x7ff2b40a2080 msgr2=0x7ff2b40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.609+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/3572136115 shutdown_connections 2023-07-02T09:00:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.609+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/3572136115 wait complete. 2023-07-02T09:00:14.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.609+0000 7ff2bc2d6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:14.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.610+0000 7ff2bc2d6700 1 Processor -- start 2023-07-02T09:00:14.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.610+0000 7ff2bc2d6700 1 -- start start 2023-07-02T09:00:14.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.611+0000 7ff2bc2d6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4132f50 0x7ff2b41ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:14.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.611+0000 7ff2bc2d6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b406d370 con 0x7ff2b4132f50 2023-07-02T09:00:14.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.611+0000 7ff2ba072700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4132f50 0x7ff2b41ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:14.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.611+0000 7ff2ba072700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4132f50 0x7ff2b41ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46214/0 (socket says 172.21.15.103:46214) 2023-07-02T09:00:14.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.611+0000 7ff2ba072700 1 -- 172.21.15.103:0/4121919108 learned_addr learned my addr 172.21.15.103:0/4121919108 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:14.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.612+0000 7ff2ba072700 1 -- 172.21.15.103:0/4121919108 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b406dde0 con 0x7ff2b4132f50 2023-07-02T09:00:14.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.612+0000 7ff2ba072700 1 --2- 172.21.15.103:0/4121919108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4132f50 0x7ff2b41ba9b0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:14.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.612+0000 7ff2aaffd700 1 -- 172.21.15.103:0/4121919108 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff2a4004b80 con 0x7ff2b4132f50 2023-07-02T09:00:14.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.612+0000 7ff2aaffd700 1 -- 172.21.15.103:0/4121919108 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff2a4004ce0 con 0x7ff2b4132f50 2023-07-02T09:00:14.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.612+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b41ba4f0 con 0x7ff2b4132f50 2023-07-02T09:00:14.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.612+0000 7ff2aaffd700 1 -- 172.21.15.103:0/4121919108 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff2a400c4f0 con 0x7ff2b4132f50 2023-07-02T09:00:14.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.612+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b4071e40 con 0x7ff2b4132f50 2023-07-02T09:00:14.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.613+0000 7ff2aaffd700 1 -- 172.21.15.103:0/4121919108 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ff2a400c650 con 0x7ff2b4132f50 2023-07-02T09:00:14.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.613+0000 7ff2aaffd700 1 --2- 172.21.15.103:0/4121919108 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff2980377c0 0x7ff298039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:14.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.613+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b4061ea0 con 0x7ff2b4132f50 2023-07-02T09:00:14.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.613+0000 7ff2aaffd700 1 -- 172.21.15.103:0/4121919108 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ff2a4046970 con 0x7ff2b4132f50 2023-07-02T09:00:14.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.613+0000 7ff2b9871700 1 --2- 172.21.15.103:0/4121919108 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff2980377c0 0x7ff298039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:14.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.614+0000 7ff2b9871700 1 --2- 172.21.15.103:0/4121919108 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff2980377c0 0x7ff298039cd0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:14.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.618+0000 7ff2aaffd700 1 -- 172.21.15.103:0/4121919108 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ff2a4009c80 con 0x7ff2b4132f50 2023-07-02T09:00:14.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.811+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff2b40725b0 con 0x7ff2b4132f50 2023-07-02T09:00:14.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.811+0000 7ff2aaffd700 1 -- 172.21.15.103:0/4121919108 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ff2a402c020 con 0x7ff2b4132f50 2023-07-02T09:00:14.812 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:14.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.813+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff2980377c0 msgr2=0x7ff298039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:14.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.813+0000 7ff2bc2d6700 1 --2- 172.21.15.103:0/4121919108 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff2980377c0 0x7ff298039cd0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:14.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.813+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4132f50 msgr2=0x7ff2b41ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:14.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.813+0000 7ff2bc2d6700 1 --2- 172.21.15.103:0/4121919108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4132f50 0x7ff2b41ba9b0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:14.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.813+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 shutdown_connections 2023-07-02T09:00:14.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.813+0000 7ff2bc2d6700 1 --2- 172.21.15.103:0/4121919108 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff2980377c0 0x7ff298039cd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:14.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.813+0000 7ff2bc2d6700 1 --2- 172.21.15.103:0/4121919108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b4132f50 0x7ff2b41ba9b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:14.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.813+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 >> 172.21.15.103:0/4121919108 conn(0x7ff2b40a2080 msgr2=0x7ff2b40a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:14.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.814+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 shutdown_connections 2023-07-02T09:00:14.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:14.814+0000 7ff2bc2d6700 1 -- 172.21.15.103:0/4121919108 wait complete. 2023-07-02T09:00:14.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:15.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:15.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.960+0000 7f9c68c64700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:15.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.965+0000 7f9c68c64700 1 Processor -- start 2023-07-02T09:00:15.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.965+0000 7f9c68c64700 1 -- start start 2023-07-02T09:00:15.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c68c64700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9c64097520 0x7f9c64097930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:15.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c68c64700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c64071cc0 con 0x7f9c64130570 2023-07-02T09:00:15.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c68c64700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c64071270 con 0x7f9c64097520 2023-07-02T09:00:15.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c6259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9c64097520 0x7f9c64097930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:15.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c61d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9c64130570 0x7f9c64130980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52232/0 (socket says 172.21.15.103:52232) 2023-07-02T09:00:15.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c61d9b700 1 -- 172.21.15.103:0/3563558937 learned_addr learned my addr 172.21.15.103:0/3563558937 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:15.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c6259c700 1 -- 172.21.15.103:0/3563558937 >> v1:172.21.15.103:6789/0 conn(0x7f9c64130570 legacy=0x7f9c64130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c6259c700 1 -- 172.21.15.103:0/3563558937 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6409d1c0 con 0x7f9c64097520 2023-07-02T09:00:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c6159a700 1 -- 172.21.15.103:0/3563558937 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111948756 0 0) 0x7f9c64071cc0 con 0x7f9c64130570 2023-07-02T09:00:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c6259c700 1 --2- 172.21.15.103:0/3563558937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c64097930 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa5a71f0a6af3ea server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c6159a700 1 -- 172.21.15.103:0/3563558937 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9c58009cd0 con 0x7f9c64097520 2023-07-02T09:00:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c6159a700 1 -- 172.21.15.103:0/3563558937 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9c58009e30 con 0x7f9c64097520 2023-07-02T09:00:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.966+0000 7f9c6159a700 1 -- 172.21.15.103:0/3563558937 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9c5800e360 con 0x7f9c64097520 2023-07-02T09:00:15.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.967+0000 7f9c68c64700 1 -- 172.21.15.103:0/3563558937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 msgr2=0x7f9c64097930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:15.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.967+0000 7f9c68c64700 1 --2- 172.21.15.103:0/3563558937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c64097930 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:15.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.967+0000 7f9c68c64700 1 -- 172.21.15.103:0/3563558937 shutdown_connections 2023-07-02T09:00:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.967+0000 7f9c68c64700 1 --2- 172.21.15.103:0/3563558937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c64097930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.967+0000 7f9c68c64700 1 -- 172.21.15.103:0/3563558937 >> 172.21.15.103:0/3563558937 conn(0x7f9c640a1030 msgr2=0x7f9c640a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.967+0000 7f9c68c64700 1 -- 172.21.15.103:0/3563558937 shutdown_connections 2023-07-02T09:00:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.967+0000 7f9c68c64700 1 -- 172.21.15.103:0/3563558937 wait complete. 2023-07-02T09:00:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.967+0000 7f9c68c64700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.968+0000 7f9c68c64700 1 Processor -- start 2023-07-02T09:00:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.969+0000 7f9c68c64700 1 -- start start 2023-07-02T09:00:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.969+0000 7f9c68c64700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c6406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.969+0000 7f9c68c64700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64094a20 con 0x7f9c64097520 2023-07-02T09:00:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.969+0000 7f9c6259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c6406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.969+0000 7f9c6259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c6406d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46232/0 (socket says 172.21.15.103:46232) 2023-07-02T09:00:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.969+0000 7f9c6259c700 1 -- 172.21.15.103:0/1043804079 learned_addr learned my addr 172.21.15.103:0/1043804079 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.970+0000 7f9c6259c700 1 -- 172.21.15.103:0/1043804079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c64094c50 con 0x7f9c64097520 2023-07-02T09:00:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.970+0000 7f9c6259c700 1 --2- 172.21.15.103:0/1043804079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c6406d540 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:15.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.970+0000 7f9c52ffd700 1 -- 172.21.15.103:0/1043804079 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9c58014540 con 0x7f9c64097520 2023-07-02T09:00:15.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.970+0000 7f9c52ffd700 1 -- 172.21.15.103:0/1043804079 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9c580146a0 con 0x7f9c64097520 2023-07-02T09:00:15.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.970+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6412dcf0 con 0x7f9c64097520 2023-07-02T09:00:15.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.970+0000 7f9c52ffd700 1 -- 172.21.15.103:0/1043804079 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9c58014940 con 0x7f9c64097520 2023-07-02T09:00:15.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.970+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6412cf40 con 0x7f9c64097520 2023-07-02T09:00:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.971+0000 7f9c52ffd700 1 -- 172.21.15.103:0/1043804079 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f9c58014af0 con 0x7f9c64097520 2023-07-02T09:00:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.971+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c64061ea0 con 0x7f9c64097520 2023-07-02T09:00:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.971+0000 7f9c52ffd700 1 --2- 172.21.15.103:0/1043804079 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c440377c0 0x7f9c44039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.971+0000 7f9c52ffd700 1 -- 172.21.15.103:0/1043804079 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9c580484a0 con 0x7f9c64097520 2023-07-02T09:00:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.971+0000 7f9c61d9b700 1 --2- 172.21.15.103:0/1043804079 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c440377c0 0x7f9c44039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.972+0000 7f9c61d9b700 1 --2- 172.21.15.103:0/1043804079 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c440377c0 0x7f9c44039cd0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:15.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:15.976+0000 7f9c52ffd700 1 -- 172.21.15.103:0/1043804079 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f9c5800fe40 con 0x7f9c64097520 2023-07-02T09:00:16.169 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.169+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9c64073ca0 con 0x7f9c64097520 2023-07-02T09:00:16.169 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.169+0000 7f9c52ffd700 1 -- 172.21.15.103:0/1043804079 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f9c58011320 con 0x7f9c64097520 2023-07-02T09:00:16.170 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:16.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c440377c0 msgr2=0x7f9c44039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:16.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 --2- 172.21.15.103:0/1043804079 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c440377c0 0x7f9c44039cd0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:16.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 msgr2=0x7f9c6406d540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:16.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 --2- 172.21.15.103:0/1043804079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c6406d540 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:16.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 shutdown_connections 2023-07-02T09:00:16.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 --2- 172.21.15.103:0/1043804079 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c440377c0 0x7f9c44039cd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:16.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 --2- 172.21.15.103:0/1043804079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c64097520 0x7f9c6406d540 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:16.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 >> 172.21.15.103:0/1043804079 conn(0x7f9c640a1030 msgr2=0x7f9c6412e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:16.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.171+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 shutdown_connections 2023-07-02T09:00:16.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:16.172+0000 7f9c68c64700 1 -- 172.21.15.103:0/1043804079 wait complete. 2023-07-02T09:00:16.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:17.186 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:17.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.326+0000 7fb1aa2b2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:17.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.331+0000 7fb1aa2b2700 1 Processor -- start 2023-07-02T09:00:17.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.331+0000 7fb1aa2b2700 1 -- start start 2023-07-02T09:00:17.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.332+0000 7fb1aa2b2700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb1a4133360 0x7fb1a409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:17.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.332+0000 7fb1aa2b2700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a409e260 con 0x7fb1a4132f50 2023-07-02T09:00:17.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.332+0000 7fb1aa2b2700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a40a1640 con 0x7fb1a4133360 2023-07-02T09:00:17.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.332+0000 7fb1a37fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb1a4133360 0x7fb1a409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:17.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.332+0000 7fb1a3fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb1a4132f50 0x7fb1a409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52234/0 (socket says 172.21.15.103:52234) 2023-07-02T09:00:17.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.332+0000 7fb1a3fff700 1 -- 172.21.15.103:0/2830698061 learned_addr learned my addr 172.21.15.103:0/2830698061 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:17.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.332+0000 7fb1a37fe700 1 -- 172.21.15.103:0/2830698061 >> v1:172.21.15.103:6789/0 conn(0x7fb1a4132f50 legacy=0x7fb1a409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:17.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1a37fe700 1 -- 172.21.15.103:0/2830698061 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a4132a20 con 0x7fb1a4133360 2023-07-02T09:00:17.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1a2ffd700 1 -- 172.21.15.103:0/2830698061 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987198339 0 0) 0x7fb1a409e260 con 0x7fb1a4132f50 2023-07-02T09:00:17.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1a37fe700 1 --2- 172.21.15.103:0/2830698061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4133360 0x7fb1a409cc90 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f0e857b68ab381 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:17.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1a2ffd700 1 -- 172.21.15.103:0/2830698061 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb194009cd0 con 0x7fb1a4133360 2023-07-02T09:00:17.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1a2ffd700 1 -- 172.21.15.103:0/2830698061 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb194009e30 con 0x7fb1a4133360 2023-07-02T09:00:17.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1a2ffd700 1 -- 172.21.15.103:0/2830698061 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb19400e360 con 0x7fb1a4133360 2023-07-02T09:00:17.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/2830698061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4133360 msgr2=0x7fb1a409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:17.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1aa2b2700 1 --2- 172.21.15.103:0/2830698061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4133360 0x7fb1a409cc90 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:17.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/2830698061 shutdown_connections 2023-07-02T09:00:17.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.333+0000 7fb1aa2b2700 1 --2- 172.21.15.103:0/2830698061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4133360 0x7fb1a409cc90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:17.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.334+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/2830698061 >> 172.21.15.103:0/2830698061 conn(0x7fb1a40a2080 msgr2=0x7fb1a40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.334+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/2830698061 shutdown_connections 2023-07-02T09:00:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.334+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/2830698061 wait complete. 2023-07-02T09:00:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.334+0000 7fb1aa2b2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.335+0000 7fb1aa2b2700 1 Processor -- start 2023-07-02T09:00:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.335+0000 7fb1aa2b2700 1 -- start start 2023-07-02T09:00:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1aa2b2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4132f50 0x7fb1a41ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1aa2b2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a406d370 con 0x7fb1a4132f50 2023-07-02T09:00:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1a3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4132f50 0x7fb1a41ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:17.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1a3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4132f50 0x7fb1a41ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46254/0 (socket says 172.21.15.103:46254) 2023-07-02T09:00:17.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1a3fff700 1 -- 172.21.15.103:0/1852650462 learned_addr learned my addr 172.21.15.103:0/1852650462 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:17.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1a3fff700 1 -- 172.21.15.103:0/1852650462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a406dde0 con 0x7fb1a4132f50 2023-07-02T09:00:17.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1a3fff700 1 --2- 172.21.15.103:0/1852650462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4132f50 0x7fb1a41ba9b0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:17.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1a0ff9700 1 -- 172.21.15.103:0/1852650462 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb198004b80 con 0x7fb1a4132f50 2023-07-02T09:00:17.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1a0ff9700 1 -- 172.21.15.103:0/1852650462 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb198004ce0 con 0x7fb1a4132f50 2023-07-02T09:00:17.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a41ba4f0 con 0x7fb1a4132f50 2023-07-02T09:00:17.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1a0ff9700 1 -- 172.21.15.103:0/1852650462 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb19800c4f0 con 0x7fb1a4132f50 2023-07-02T09:00:17.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.336+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a4071e40 con 0x7fb1a4132f50 2023-07-02T09:00:17.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.337+0000 7fb1a0ff9700 1 -- 172.21.15.103:0/1852650462 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb19800c650 con 0x7fb1a4132f50 2023-07-02T09:00:17.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.337+0000 7fb1a0ff9700 1 --2- 172.21.15.103:0/1852650462 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb1900377c0 0x7fb190039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:17.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.337+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a4061ea0 con 0x7fb1a4132f50 2023-07-02T09:00:17.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.337+0000 7fb1a0ff9700 1 -- 172.21.15.103:0/1852650462 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb198046a30 con 0x7fb1a4132f50 2023-07-02T09:00:17.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.338+0000 7fb1a37fe700 1 --2- 172.21.15.103:0/1852650462 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb1900377c0 0x7fb190039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:17.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.338+0000 7fb1a37fe700 1 --2- 172.21.15.103:0/1852650462 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb1900377c0 0x7fb190039cd0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:17.342 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.342+0000 7fb1a0ff9700 1 -- 172.21.15.103:0/1852650462 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb198009c80 con 0x7fb1a4132f50 2023-07-02T09:00:17.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.535+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb1a40725b0 con 0x7fb1a4132f50 2023-07-02T09:00:17.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.535+0000 7fb1a0ff9700 1 -- 172.21.15.103:0/1852650462 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb1980098a0 con 0x7fb1a4132f50 2023-07-02T09:00:17.536 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:17.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb1900377c0 msgr2=0x7fb190039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:17.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 --2- 172.21.15.103:0/1852650462 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb1900377c0 0x7fb190039cd0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:17.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4132f50 msgr2=0x7fb1a41ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:17.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 --2- 172.21.15.103:0/1852650462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4132f50 0x7fb1a41ba9b0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:17.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 shutdown_connections 2023-07-02T09:00:17.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 --2- 172.21.15.103:0/1852650462 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb1900377c0 0x7fb190039cd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:17.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 --2- 172.21.15.103:0/1852650462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a4132f50 0x7fb1a41ba9b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:17.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 >> 172.21.15.103:0/1852650462 conn(0x7fb1a40a2080 msgr2=0x7fb1a40a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:17.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 shutdown_connections 2023-07-02T09:00:17.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:17.537+0000 7fb1aa2b2700 1 -- 172.21.15.103:0/1852650462 wait complete. 2023-07-02T09:00:17.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:18.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:18.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.691+0000 7fb8571d1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:18.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.697+0000 7fb8571d1700 1 Processor -- start 2023-07-02T09:00:18.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.697+0000 7fb8571d1700 1 -- start start 2023-07-02T09:00:18.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.697+0000 7fb8571d1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb850097520 0x7fb850097930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:18.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.697+0000 7fb8571d1700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb850071cc0 con 0x7fb850130570 2023-07-02T09:00:18.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.697+0000 7fb8571d1700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb850071270 con 0x7fb850097520 2023-07-02T09:00:18.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb854f6d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb850097520 0x7fb850097930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:18.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb84ffff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb850130570 0x7fb850130980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52236/0 (socket says 172.21.15.103:52236) 2023-07-02T09:00:18.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb84ffff700 1 -- 172.21.15.103:0/2599532245 learned_addr learned my addr 172.21.15.103:0/2599532245 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:18.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb854f6d700 1 -- 172.21.15.103:0/2599532245 >> v1:172.21.15.103:6789/0 conn(0x7fb850130570 legacy=0x7fb850130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:18.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb854f6d700 1 -- 172.21.15.103:0/2599532245 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85009d1c0 con 0x7fb850097520 2023-07-02T09:00:18.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb854f6d700 1 --2- 172.21.15.103:0/2599532245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850097930 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770523520890ee08 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:18.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb84f7fe700 1 -- 172.21.15.103:0/2599532245 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb840009cd0 con 0x7fb850097520 2023-07-02T09:00:18.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb84f7fe700 1 -- 172.21.15.103:0/2599532245 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb840009e30 con 0x7fb850097520 2023-07-02T09:00:18.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.698+0000 7fb84f7fe700 1 -- 172.21.15.103:0/2599532245 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb84000e360 con 0x7fb850097520 2023-07-02T09:00:18.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.699+0000 7fb8571d1700 1 -- 172.21.15.103:0/2599532245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 msgr2=0x7fb850097930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:18.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.699+0000 7fb8571d1700 1 --2- 172.21.15.103:0/2599532245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850097930 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:18.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.699+0000 7fb8571d1700 1 -- 172.21.15.103:0/2599532245 shutdown_connections 2023-07-02T09:00:18.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.699+0000 7fb8571d1700 1 --2- 172.21.15.103:0/2599532245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850097930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:18.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.699+0000 7fb8571d1700 1 -- 172.21.15.103:0/2599532245 >> 172.21.15.103:0/2599532245 conn(0x7fb8500a1030 msgr2=0x7fb8500a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:18.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.699+0000 7fb8571d1700 1 -- 172.21.15.103:0/2599532245 shutdown_connections 2023-07-02T09:00:18.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.699+0000 7fb8571d1700 1 -- 172.21.15.103:0/2599532245 wait complete. 2023-07-02T09:00:18.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.699+0000 7fb8571d1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:18.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.700+0000 7fb8571d1700 1 Processor -- start 2023-07-02T09:00:18.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.700+0000 7fb8571d1700 1 -- start start 2023-07-02T09:00:18.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.701+0000 7fb8571d1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850127ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:18.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.701+0000 7fb8571d1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85009a250 con 0x7fb850097520 2023-07-02T09:00:18.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb854f6d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850127ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:18.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb854f6d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850127ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46270/0 (socket says 172.21.15.103:46270) 2023-07-02T09:00:18.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb854f6d700 1 -- 172.21.15.103:0/2733289460 learned_addr learned my addr 172.21.15.103:0/2733289460 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:18.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb854f6d700 1 -- 172.21.15.103:0/2733289460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85012db00 con 0x7fb850097520 2023-07-02T09:00:18.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb854f6d700 1 --2- 172.21.15.103:0/2733289460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850127ed0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:18.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb84d7fa700 1 -- 172.21.15.103:0/2733289460 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb840013530 con 0x7fb850097520 2023-07-02T09:00:18.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb84d7fa700 1 -- 172.21.15.103:0/2733289460 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb840013690 con 0x7fb850097520 2023-07-02T09:00:18.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb850127410 con 0x7fb850097520 2023-07-02T09:00:18.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb84d7fa700 1 -- 172.21.15.103:0/2733289460 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb840013930 con 0x7fb850097520 2023-07-02T09:00:18.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.702+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb850127650 con 0x7fb850097520 2023-07-02T09:00:18.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.703+0000 7fb84d7fa700 1 -- 172.21.15.103:0/2733289460 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb840013ae0 con 0x7fb850097520 2023-07-02T09:00:18.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.703+0000 7fb84d7fa700 1 --2- 172.21.15.103:0/2733289460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb830037800 0x7fb830039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:18.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.703+0000 7fb84d7fa700 1 -- 172.21.15.103:0/2733289460 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb840018040 con 0x7fb850097520 2023-07-02T09:00:18.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.703+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85009a640 con 0x7fb850097520 2023-07-02T09:00:18.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.704+0000 7fb84ffff700 1 --2- 172.21.15.103:0/2733289460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb830037800 0x7fb830039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:18.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.704+0000 7fb84ffff700 1 --2- 172.21.15.103:0/2733289460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb830037800 0x7fb830039d10 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.708+0000 7fb84d7fa700 1 -- 172.21.15.103:0/2733289460 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb840018e00 con 0x7fb850097520 2023-07-02T09:00:18.901 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.901+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb850061ea0 con 0x7fb850097520 2023-07-02T09:00:18.901 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.901+0000 7fb84d7fa700 1 -- 172.21.15.103:0/2733289460 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb8400189f0 con 0x7fb850097520 2023-07-02T09:00:18.902 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:18.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~33s 2023-07-02T09:00:18.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.903+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb830037800 msgr2=0x7fb830039d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:18.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.903+0000 7fb8571d1700 1 --2- 172.21.15.103:0/2733289460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb830037800 0x7fb830039d10 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:18.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.903+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 msgr2=0x7fb850127ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:18.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.904+0000 7fb8571d1700 1 --2- 172.21.15.103:0/2733289460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850127ed0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:18.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.904+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 shutdown_connections 2023-07-02T09:00:18.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.904+0000 7fb8571d1700 1 --2- 172.21.15.103:0/2733289460 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb830037800 0x7fb830039d10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:18.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.904+0000 7fb8571d1700 1 --2- 172.21.15.103:0/2733289460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb850097520 0x7fb850127ed0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:18.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.904+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 >> 172.21.15.103:0/2733289460 conn(0x7fb8500a1030 msgr2=0x7fb85012e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:18.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.904+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 shutdown_connections 2023-07-02T09:00:18.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:18.904+0000 7fb8571d1700 1 -- 172.21.15.103:0/2733289460 wait complete. 2023-07-02T09:00:18.917 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:19.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:20.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.046+0000 7f2cad4e9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:20.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.051+0000 7f2cad4e9700 1 Processor -- start 2023-07-02T09:00:20.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.051+0000 7f2cad4e9700 1 -- start start 2023-07-02T09:00:20.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2cad4e9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2ca8133360 0x7f2ca809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:20.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2cad4e9700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca809e260 con 0x7f2ca8132f50 2023-07-02T09:00:20.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2cad4e9700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca80a1640 con 0x7f2ca8133360 2023-07-02T09:00:20.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2ca67fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2ca8133360 0x7f2ca809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2ca6ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2ca8132f50 0x7f2ca809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52248/0 (socket says 172.21.15.103:52248) 2023-07-02T09:00:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2ca6ffd700 1 -- 172.21.15.103:0/3908960785 learned_addr learned my addr 172.21.15.103:0/3908960785 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2ca67fc700 1 -- 172.21.15.103:0/3908960785 >> v1:172.21.15.103:6789/0 conn(0x7f2ca8132f50 legacy=0x7f2ca809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2ca67fc700 1 -- 172.21.15.103:0/3908960785 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca8132a20 con 0x7f2ca8133360 2023-07-02T09:00:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2ca5ffb700 1 -- 172.21.15.103:0/3908960785 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975251455 0 0) 0x7f2ca809e260 con 0x7f2ca8132f50 2023-07-02T09:00:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.052+0000 7f2ca67fc700 1 --2- 172.21.15.103:0/3908960785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8133360 0x7f2ca809cc90 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e924fdff52c80d3 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2ca5ffb700 1 -- 172.21.15.103:0/3908960785 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2c90009cd0 con 0x7f2ca8133360 2023-07-02T09:00:20.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2ca5ffb700 1 -- 172.21.15.103:0/3908960785 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2c90009e30 con 0x7f2ca8133360 2023-07-02T09:00:20.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2ca5ffb700 1 -- 172.21.15.103:0/3908960785 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2c9000e360 con 0x7f2ca8133360 2023-07-02T09:00:20.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2cad4e9700 1 -- 172.21.15.103:0/3908960785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8133360 msgr2=0x7f2ca809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:20.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2cad4e9700 1 --2- 172.21.15.103:0/3908960785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8133360 0x7f2ca809cc90 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:20.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2cad4e9700 1 -- 172.21.15.103:0/3908960785 shutdown_connections 2023-07-02T09:00:20.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2cad4e9700 1 --2- 172.21.15.103:0/3908960785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8133360 0x7f2ca809cc90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:20.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2cad4e9700 1 -- 172.21.15.103:0/3908960785 >> 172.21.15.103:0/3908960785 conn(0x7f2ca80a2080 msgr2=0x7f2ca80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:20.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2cad4e9700 1 -- 172.21.15.103:0/3908960785 shutdown_connections 2023-07-02T09:00:20.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.053+0000 7f2cad4e9700 1 -- 172.21.15.103:0/3908960785 wait complete. 2023-07-02T09:00:20.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.054+0000 7f2cad4e9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:20.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.055+0000 7f2cad4e9700 1 Processor -- start 2023-07-02T09:00:20.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.055+0000 7f2cad4e9700 1 -- start start 2023-07-02T09:00:20.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.055+0000 7f2cad4e9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8132f50 0x7f2ca812c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:20.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.055+0000 7f2cad4e9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca806d2f0 con 0x7f2ca8132f50 2023-07-02T09:00:20.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.055+0000 7f2ca6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8132f50 0x7f2ca812c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:20.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.055+0000 7f2ca6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8132f50 0x7f2ca812c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46288/0 (socket says 172.21.15.103:46288) 2023-07-02T09:00:20.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.055+0000 7f2ca6ffd700 1 -- 172.21.15.103:0/805851229 learned_addr learned my addr 172.21.15.103:0/805851229 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:20.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.056+0000 7f2ca6ffd700 1 -- 172.21.15.103:0/805851229 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca806dd60 con 0x7f2ca8132f50 2023-07-02T09:00:20.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.056+0000 7f2ca6ffd700 1 --2- 172.21.15.103:0/805851229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8132f50 0x7f2ca812c1a0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:20.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.056+0000 7f2c9f7fe700 1 -- 172.21.15.103:0/805851229 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2c98004b80 con 0x7f2ca8132f50 2023-07-02T09:00:20.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.056+0000 7f2c9f7fe700 1 -- 172.21.15.103:0/805851229 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2c98004ce0 con 0x7f2ca8132f50 2023-07-02T09:00:20.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.056+0000 7f2c9f7fe700 1 -- 172.21.15.103:0/805851229 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2c9800c4f0 con 0x7f2ca8132f50 2023-07-02T09:00:20.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.056+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca812bc50 con 0x7f2ca8132f50 2023-07-02T09:00:20.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.056+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca806f850 con 0x7f2ca8132f50 2023-07-02T09:00:20.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.057+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca806dbb0 con 0x7f2ca8132f50 2023-07-02T09:00:20.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.057+0000 7f2c9f7fe700 1 -- 172.21.15.103:0/805851229 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2c9800c650 con 0x7f2ca8132f50 2023-07-02T09:00:20.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.057+0000 7f2c9f7fe700 1 --2- 172.21.15.103:0/805851229 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2c880377c0 0x7f2c88039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:20.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.057+0000 7f2c9f7fe700 1 -- 172.21.15.103:0/805851229 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2c98046a30 con 0x7f2ca8132f50 2023-07-02T09:00:20.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.057+0000 7f2ca67fc700 1 --2- 172.21.15.103:0/805851229 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2c880377c0 0x7f2c88039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:20.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.058+0000 7f2ca67fc700 1 --2- 172.21.15.103:0/805851229 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2c880377c0 0x7f2c88039cd0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:20.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.062+0000 7f2c9f7fe700 1 -- 172.21.15.103:0/805851229 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2c98009c80 con 0x7f2ca8132f50 2023-07-02T09:00:20.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.253+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2ca8061ea0 con 0x7f2ca8132f50 2023-07-02T09:00:20.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.253+0000 7f2c9f7fe700 1 -- 172.21.15.103:0/805851229 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2c980098a0 con 0x7f2ca8132f50 2023-07-02T09:00:20.254 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:20.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2c880377c0 msgr2=0x7f2c88039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:20.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 --2- 172.21.15.103:0/805851229 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2c880377c0 0x7f2c88039cd0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:20.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8132f50 msgr2=0x7f2ca812c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:20.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 --2- 172.21.15.103:0/805851229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8132f50 0x7f2ca812c1a0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:20.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 shutdown_connections 2023-07-02T09:00:20.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 --2- 172.21.15.103:0/805851229 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2c880377c0 0x7f2c88039cd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:20.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 --2- 172.21.15.103:0/805851229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ca8132f50 0x7f2ca812c1a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:20.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 >> 172.21.15.103:0/805851229 conn(0x7f2ca80a2080 msgr2=0x7f2ca8130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:20.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 shutdown_connections 2023-07-02T09:00:20.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:20.255+0000 7f2cad4e9700 1 -- 172.21.15.103:0/805851229 wait complete. 2023-07-02T09:00:20.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:21.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:21.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.405+0000 7f9694c76700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:21.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.410+0000 7f9694c76700 1 Processor -- start 2023-07-02T09:00:21.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.410+0000 7f9694c76700 1 -- start start 2023-07-02T09:00:21.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.411+0000 7f9694c76700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f969009eb40 0x7f969009d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:21.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.411+0000 7f9694c76700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969009ce10 con 0x7f969009d580 2023-07-02T09:00:21.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.411+0000 7f9694c76700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969006d3d0 con 0x7f969009eb40 2023-07-02T09:00:21.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.411+0000 7f968e59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f969009eb40 0x7f969009d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:21.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.411+0000 7f968e59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f969009eb40 0x7f969009d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46298/0 (socket says 172.21.15.103:46298) 2023-07-02T09:00:21.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.411+0000 7f968e59c700 1 -- 172.21.15.103:0/1410103596 learned_addr learned my addr 172.21.15.103:0/1410103596 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:21.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.411+0000 7f968dd9b700 1 --1- 172.21.15.103:0/1410103596 >> v1:172.21.15.103:6789/0 conn(0x7f969009d580 0x7f9690132920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52264/0 (socket says 172.21.15.103:52264) 2023-07-02T09:00:21.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.412+0000 7f968e59c700 1 -- 172.21.15.103:0/1410103596 >> v1:172.21.15.103:6789/0 conn(0x7f969009d580 legacy=0x7f9690132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:21.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.412+0000 7f968e59c700 1 -- 172.21.15.103:0/1410103596 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969009f2f0 con 0x7f969009eb40 2023-07-02T09:00:21.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.412+0000 7f968d59a700 1 -- 172.21.15.103:0/1410103596 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760430504 0 0) 0x7f969009ce10 con 0x7f969009d580 2023-07-02T09:00:21.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.412+0000 7f968e59c700 1 --2- 172.21.15.103:0/1410103596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009eb40 0x7f969009d040 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf5c2050d561588 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.412+0000 7f968d59a700 1 -- 172.21.15.103:0/1410103596 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9684009d90 con 0x7f969009eb40 2023-07-02T09:00:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.412+0000 7f968d59a700 1 -- 172.21.15.103:0/1410103596 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f968400e230 con 0x7f969009eb40 2023-07-02T09:00:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.412+0000 7f968d59a700 1 -- 172.21.15.103:0/1410103596 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f968400e4d0 con 0x7f969009eb40 2023-07-02T09:00:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.413+0000 7f9694c76700 1 -- 172.21.15.103:0/1410103596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009eb40 msgr2=0x7f969009d040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.413+0000 7f9694c76700 1 --2- 172.21.15.103:0/1410103596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009eb40 0x7f969009d040 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.413+0000 7f9694c76700 1 -- 172.21.15.103:0/1410103596 shutdown_connections 2023-07-02T09:00:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.413+0000 7f9694c76700 1 --2- 172.21.15.103:0/1410103596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009eb40 0x7f969009d040 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.413+0000 7f9694c76700 1 -- 172.21.15.103:0/1410103596 >> 172.21.15.103:0/1410103596 conn(0x7f96900a1a80 msgr2=0x7f96900a1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.413+0000 7f9694c76700 1 -- 172.21.15.103:0/1410103596 shutdown_connections 2023-07-02T09:00:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.413+0000 7f9694c76700 1 -- 172.21.15.103:0/1410103596 wait complete. 2023-07-02T09:00:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.413+0000 7f9694c76700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.414+0000 7f9694c76700 1 Processor -- start 2023-07-02T09:00:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.414+0000 7f9694c76700 1 -- start start 2023-07-02T09:00:21.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f9694c76700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009d580 0x7f969011f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:21.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f9694c76700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9690121820 con 0x7f969009d580 2023-07-02T09:00:21.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f968e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009d580 0x7f969011f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f968e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009d580 0x7f969011f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46310/0 (socket says 172.21.15.103:46310) 2023-07-02T09:00:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f968e59c700 1 -- 172.21.15.103:0/967061365 learned_addr learned my addr 172.21.15.103:0/967061365 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f968e59c700 1 -- 172.21.15.103:0/967061365 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969011f470 con 0x7f969009d580 2023-07-02T09:00:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f968e59c700 1 --2- 172.21.15.103:0/967061365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009d580 0x7f969011f790 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f967effd700 1 -- 172.21.15.103:0/967061365 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9684014540 con 0x7f969009d580 2023-07-02T09:00:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f967effd700 1 -- 172.21.15.103:0/967061365 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f96840146a0 con 0x7f969009d580 2023-07-02T09:00:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.415+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96901209b0 con 0x7f969009d580 2023-07-02T09:00:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.416+0000 7f967effd700 1 -- 172.21.15.103:0/967061365 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9684014940 con 0x7f969009d580 2023-07-02T09:00:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.416+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9690121290 con 0x7f969009d580 2023-07-02T09:00:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.416+0000 7f967effd700 1 -- 172.21.15.103:0/967061365 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f9684014af0 con 0x7f969009d580 2023-07-02T09:00:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.416+0000 7f967effd700 1 --2- 172.21.15.103:0/967061365 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9670037810 0x7f9670039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.416+0000 7f967effd700 1 -- 172.21.15.103:0/967061365 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9684048430 con 0x7f969009d580 2023-07-02T09:00:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.417+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690061ea0 con 0x7f969009d580 2023-07-02T09:00:21.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.417+0000 7f968dd9b700 1 --2- 172.21.15.103:0/967061365 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9670037810 0x7f9670039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:21.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.417+0000 7f968dd9b700 1 --2- 172.21.15.103:0/967061365 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9670037810 0x7f9670039d20 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:21.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.421+0000 7f967effd700 1 -- 172.21.15.103:0/967061365 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f968400fc80 con 0x7f969009d580 2023-07-02T09:00:21.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.617+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f96900724f0 con 0x7f969009d580 2023-07-02T09:00:21.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.617+0000 7f967effd700 1 -- 172.21.15.103:0/967061365 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f968400fa50 con 0x7f969009d580 2023-07-02T09:00:21.618 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:21.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9670037810 msgr2=0x7f9670039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:21.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 --2- 172.21.15.103:0/967061365 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9670037810 0x7f9670039d20 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:21.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009d580 msgr2=0x7f969011f790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:21.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 --2- 172.21.15.103:0/967061365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009d580 0x7f969011f790 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:21.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 shutdown_connections 2023-07-02T09:00:21.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 --2- 172.21.15.103:0/967061365 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9670037810 0x7f9670039d20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:21.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 --2- 172.21.15.103:0/967061365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f969009d580 0x7f969011f790 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:21.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 >> 172.21.15.103:0/967061365 conn(0x7f96900a1a80 msgr2=0x7f9690131f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:21.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 shutdown_connections 2023-07-02T09:00:21.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:21.619+0000 7f9694c76700 1 -- 172.21.15.103:0/967061365 wait complete. 2023-07-02T09:00:21.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:22.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:22.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.773+0000 7f4559574700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:22.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.778+0000 7f4559574700 1 Processor -- start 2023-07-02T09:00:22.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.778+0000 7f4559574700 1 -- start start 2023-07-02T09:00:22.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.779+0000 7f4559574700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f455412f250 0x7f455412f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:22.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.779+0000 7f4559574700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455406df60 con 0x7f4554098a00 2023-07-02T09:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.779+0000 7f4559574700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f455409a710 con 0x7f455412f250 2023-07-02T09:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.779+0000 7f4552ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f455412f250 0x7f455412f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.779+0000 7f45527fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4554098a00 0x7f4554098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:52268/0 (socket says 172.21.15.103:52268) 2023-07-02T09:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.779+0000 7f45527fc700 1 -- 172.21.15.103:0/2767692171 learned_addr learned my addr 172.21.15.103:0/2767692171 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.779+0000 7f4552ffd700 1 -- 172.21.15.103:0/2767692171 >> v1:172.21.15.103:6789/0 conn(0x7f4554098a00 legacy=0x7f4554098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.779+0000 7f4552ffd700 1 -- 172.21.15.103:0/2767692171 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455412ed20 con 0x7f455412f250 2023-07-02T09:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4552ffd700 1 --2- 172.21.15.103:0/2767692171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f455412f250 0x7f455412f660 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78744d63d53a4eb5 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4551ffb700 1 -- 172.21.15.103:0/2767692171 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4548009cd0 con 0x7f455412f250 2023-07-02T09:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4551ffb700 1 -- 172.21.15.103:0/2767692171 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4548009e30 con 0x7f455412f250 2023-07-02T09:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4551ffb700 1 -- 172.21.15.103:0/2767692171 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f454800f470 con 0x7f455412f250 2023-07-02T09:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4559574700 1 -- 172.21.15.103:0/2767692171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f455412f250 msgr2=0x7f455412f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4559574700 1 --2- 172.21.15.103:0/2767692171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f455412f250 0x7f455412f660 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4559574700 1 -- 172.21.15.103:0/2767692171 shutdown_connections 2023-07-02T09:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4559574700 1 --2- 172.21.15.103:0/2767692171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f455412f250 0x7f455412f660 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.780+0000 7f4559574700 1 -- 172.21.15.103:0/2767692171 >> 172.21.15.103:0/2767692171 conn(0x7f455409d810 msgr2=0x7f455409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.781+0000 7f4559574700 1 -- 172.21.15.103:0/2767692171 shutdown_connections 2023-07-02T09:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.781+0000 7f4559574700 1 -- 172.21.15.103:0/2767692171 wait complete. 2023-07-02T09:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.781+0000 7f4559574700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.782+0000 7f4559574700 1 Processor -- start 2023-07-02T09:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.782+0000 7f4559574700 1 -- start start 2023-07-02T09:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.782+0000 7f4559574700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4554098a00 0x7f4554117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.782+0000 7f4559574700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455411f940 con 0x7f4554098a00 2023-07-02T09:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f4552ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4554098a00 0x7f4554117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f4552ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4554098a00 0x7f4554117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46322/0 (socket says 172.21.15.103:46322) 2023-07-02T09:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f4552ffd700 1 -- 172.21.15.103:0/630216572 learned_addr learned my addr 172.21.15.103:0/630216572 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f4552ffd700 1 -- 172.21.15.103:0/630216572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4554129080 con 0x7f4554098a00 2023-07-02T09:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f4552ffd700 1 --2- 172.21.15.103:0/630216572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4554098a00 0x7f4554117360 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f453b7fe700 1 -- 172.21.15.103:0/630216572 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4548011420 con 0x7f4554098a00 2023-07-02T09:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f453b7fe700 1 -- 172.21.15.103:0/630216572 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4548011580 con 0x7f4554098a00 2023-07-02T09:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4554118f50 con 0x7f4554098a00 2023-07-02T09:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f453b7fe700 1 -- 172.21.15.103:0/630216572 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4548011820 con 0x7f4554098a00 2023-07-02T09:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.783+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455411f610 con 0x7f4554098a00 2023-07-02T09:00:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.784+0000 7f453b7fe700 1 -- 172.21.15.103:0/630216572 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f45480119d0 con 0x7f4554098a00 2023-07-02T09:00:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.784+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4554061ea0 con 0x7f4554098a00 2023-07-02T09:00:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.784+0000 7f453b7fe700 1 --2- 172.21.15.103:0/630216572 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f45400377c0 0x7f4540039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.784+0000 7f453b7fe700 1 -- 172.21.15.103:0/630216572 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4548017040 con 0x7f4554098a00 2023-07-02T09:00:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.785+0000 7f45527fc700 1 --2- 172.21.15.103:0/630216572 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f45400377c0 0x7f4540039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.785+0000 7f45527fc700 1 --2- 172.21.15.103:0/630216572 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f45400377c0 0x7f4540039cd0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.789+0000 7f453b7fe700 1 -- 172.21.15.103:0/630216572 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f454800c560 con 0x7f4554098a00 2023-07-02T09:00:22.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.984+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f455411f610 con 0x7f4554098a00 2023-07-02T09:00:22.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.985+0000 7f453b7fe700 1 -- 172.21.15.103:0/630216572 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f455411f610 con 0x7f4554098a00 2023-07-02T09:00:22.985 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:22.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f45400377c0 msgr2=0x7f4540039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:22.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 --2- 172.21.15.103:0/630216572 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f45400377c0 0x7f4540039cd0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:22.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4554098a00 msgr2=0x7f4554117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:22.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 --2- 172.21.15.103:0/630216572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4554098a00 0x7f4554117360 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:22.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 shutdown_connections 2023-07-02T09:00:22.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 --2- 172.21.15.103:0/630216572 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f45400377c0 0x7f4540039cd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:22.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 --2- 172.21.15.103:0/630216572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4554098a00 0x7f4554117360 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:22.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 >> 172.21.15.103:0/630216572 conn(0x7f455409d810 msgr2=0x7f455409f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:22.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 shutdown_connections 2023-07-02T09:00:22.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:22.987+0000 7f4559574700 1 -- 172.21.15.103:0/630216572 wait complete. 2023-07-02T09:00:23.000 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:24.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:24.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.137+0000 7f5c129ba700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:24.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.142+0000 7f5c129ba700 1 Processor -- start 2023-07-02T09:00:24.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.142+0000 7f5c129ba700 1 -- start start 2023-07-02T09:00:24.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c129ba700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5c0c132f50 0x7f5c0c133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:24.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c129ba700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0c09e260 con 0x7f5c0c09c580 2023-07-02T09:00:24.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c129ba700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0c0a1640 con 0x7f5c0c132f50 2023-07-02T09:00:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5c0c132f50 0x7f5c0c133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0b7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f5c0c09c580 0x7f5c0c09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47982/0 (socket says 172.21.15.103:47982) 2023-07-02T09:00:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5c0c132f50 0x7f5c0c133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44208/0 (socket says 172.21.15.103:44208) 2023-07-02T09:00:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0b7fe700 1 -- 172.21.15.103:0/1993768338 learned_addr learned my addr 172.21.15.103:0/1993768338 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0bfff700 1 -- 172.21.15.103:0/1993768338 >> v1:172.21.15.103:6789/0 conn(0x7f5c0c09c580 legacy=0x7f5c0c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0bfff700 1 -- 172.21.15.103:0/1993768338 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c132a20 con 0x7f5c0c132f50 2023-07-02T09:00:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0bfff700 1 --2- 172.21.15.103:0/1993768338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c132f50 0x7f5c0c133360 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d73df2b2eb2d979 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0affd700 1 -- 172.21.15.103:0/1993768338 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5c00009cd0 con 0x7f5c0c132f50 2023-07-02T09:00:24.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0affd700 1 -- 172.21.15.103:0/1993768338 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5c00009e30 con 0x7f5c0c132f50 2023-07-02T09:00:24.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.143+0000 7f5c0affd700 1 -- 172.21.15.103:0/1993768338 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5c0000e360 con 0x7f5c0c132f50 2023-07-02T09:00:24.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.144+0000 7f5c129ba700 1 -- 172.21.15.103:0/1993768338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c132f50 msgr2=0x7f5c0c133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:24.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.144+0000 7f5c129ba700 1 --2- 172.21.15.103:0/1993768338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c132f50 0x7f5c0c133360 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:24.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.144+0000 7f5c129ba700 1 -- 172.21.15.103:0/1993768338 shutdown_connections 2023-07-02T09:00:24.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.144+0000 7f5c129ba700 1 --2- 172.21.15.103:0/1993768338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c132f50 0x7f5c0c133360 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:24.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.144+0000 7f5c129ba700 1 -- 172.21.15.103:0/1993768338 >> 172.21.15.103:0/1993768338 conn(0x7f5c0c0a2080 msgr2=0x7f5c0c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:24.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.144+0000 7f5c129ba700 1 -- 172.21.15.103:0/1993768338 shutdown_connections 2023-07-02T09:00:24.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.144+0000 7f5c129ba700 1 -- 172.21.15.103:0/1993768338 wait complete. 2023-07-02T09:00:24.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.144+0000 7f5c129ba700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:24.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.145+0000 7f5c129ba700 1 Processor -- start 2023-07-02T09:00:24.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.145+0000 7f5c129ba700 1 -- start start 2023-07-02T09:00:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.146+0000 7f5c129ba700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c09c580 0x7f5c0c12b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.146+0000 7f5c129ba700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c06f9d0 con 0x7f5c0c09c580 2023-07-02T09:00:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.146+0000 7f5c0bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c09c580 0x7f5c0c12b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.146+0000 7f5c0bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c09c580 0x7f5c0c12b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44216/0 (socket says 172.21.15.103:44216) 2023-07-02T09:00:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.146+0000 7f5c0bfff700 1 -- 172.21.15.103:0/321972110 learned_addr learned my addr 172.21.15.103:0/321972110 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.146+0000 7f5c0bfff700 1 -- 172.21.15.103:0/321972110 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c133a00 con 0x7f5c0c09c580 2023-07-02T09:00:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.146+0000 7f5c0bfff700 1 --2- 172.21.15.103:0/321972110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c09c580 0x7f5c0c12b7b0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.147+0000 7f5c08ff9700 1 -- 172.21.15.103:0/321972110 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5c00013530 con 0x7f5c0c09c580 2023-07-02T09:00:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.147+0000 7f5c08ff9700 1 -- 172.21.15.103:0/321972110 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5c00013690 con 0x7f5c0c09c580 2023-07-02T09:00:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.147+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c0c06d160 con 0x7f5c0c09c580 2023-07-02T09:00:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.147+0000 7f5c08ff9700 1 -- 172.21.15.103:0/321972110 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5c00013930 con 0x7f5c0c09c580 2023-07-02T09:00:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.147+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0c06da70 con 0x7f5c0c09c580 2023-07-02T09:00:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.148+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0c061ea0 con 0x7f5c0c09c580 2023-07-02T09:00:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.148+0000 7f5c08ff9700 1 -- 172.21.15.103:0/321972110 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f5c00013ae0 con 0x7f5c0c09c580 2023-07-02T09:00:24.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.148+0000 7f5c08ff9700 1 --2- 172.21.15.103:0/321972110 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5bf80377c0 0x7f5bf8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:24.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.148+0000 7f5c08ff9700 1 -- 172.21.15.103:0/321972110 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f5c00018040 con 0x7f5c0c09c580 2023-07-02T09:00:24.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.148+0000 7f5c0b7fe700 1 --2- 172.21.15.103:0/321972110 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5bf80377c0 0x7f5bf8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:24.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.148+0000 7f5c0b7fe700 1 --2- 172.21.15.103:0/321972110 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5bf80377c0 0x7f5bf8039cd0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:24.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.152+0000 7f5c08ff9700 1 -- 172.21.15.103:0/321972110 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f5c00018e00 con 0x7f5c0c09c580 2023-07-02T09:00:24.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.344+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5c0c0725b0 con 0x7f5c0c09c580 2023-07-02T09:00:24.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.345+0000 7f5c08ff9700 1 -- 172.21.15.103:0/321972110 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f5c00010e60 con 0x7f5c0c09c580 2023-07-02T09:00:24.345 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:24.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5bf80377c0 msgr2=0x7f5bf8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:24.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 --2- 172.21.15.103:0/321972110 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5bf80377c0 0x7f5bf8039cd0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:24.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c09c580 msgr2=0x7f5c0c12b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:24.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 --2- 172.21.15.103:0/321972110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c09c580 0x7f5c0c12b7b0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:24.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 shutdown_connections 2023-07-02T09:00:24.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 --2- 172.21.15.103:0/321972110 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5bf80377c0 0x7f5bf8039cd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:24.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 --2- 172.21.15.103:0/321972110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c09c580 0x7f5c0c12b7b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:24.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 >> 172.21.15.103:0/321972110 conn(0x7f5c0c0a2080 msgr2=0x7f5c0c1babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:24.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 shutdown_connections 2023-07-02T09:00:24.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:24.347+0000 7f5c129ba700 1 -- 172.21.15.103:0/321972110 wait complete. 2023-07-02T09:00:24.360 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:24.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~39s 2023-07-02T09:00:25.361 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:25.504 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.500+0000 7f7c47318700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:25.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.505+0000 7f7c47318700 1 Processor -- start 2023-07-02T09:00:25.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.505+0000 7f7c47318700 1 -- start start 2023-07-02T09:00:25.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.506+0000 7f7c47318700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7c40132030 0x7f7c40134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:25.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.506+0000 7f7c47318700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4006df60 con 0x7f7c4012f250 2023-07-02T09:00:25.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.506+0000 7f7c47318700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4009a710 con 0x7f7c40132030 2023-07-02T09:00:25.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.506+0000 7f7c448b3700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7c40132030 0x7f7c40134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:25.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.506+0000 7f7c450b4700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7c4012f250 0x7f7c4012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47984/0 (socket says 172.21.15.103:47984) 2023-07-02T09:00:25.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.506+0000 7f7c450b4700 1 -- 172.21.15.103:0/1818929647 learned_addr learned my addr 172.21.15.103:0/1818929647 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:25.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.506+0000 7f7c448b3700 1 -- 172.21.15.103:0/1818929647 >> v1:172.21.15.103:6789/0 conn(0x7f7c4012f250 legacy=0x7f7c4012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:25.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.506+0000 7f7c448b3700 1 -- 172.21.15.103:0/1818929647 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4012ed20 con 0x7f7c40132030 2023-07-02T09:00:25.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c448b3700 1 --2- 172.21.15.103:0/1818929647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c40132030 0x7f7c40134450 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd30d61a2db14902 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:25.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c37fff700 1 -- 172.21.15.103:0/1818929647 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7c3c009cd0 con 0x7f7c40132030 2023-07-02T09:00:25.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c37fff700 1 -- 172.21.15.103:0/1818929647 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7c3c009e30 con 0x7f7c40132030 2023-07-02T09:00:25.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c37fff700 1 -- 172.21.15.103:0/1818929647 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7c3c00f470 con 0x7f7c40132030 2023-07-02T09:00:25.510 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c47318700 1 -- 172.21.15.103:0/1818929647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c40132030 msgr2=0x7f7c40134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:25.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c47318700 1 --2- 172.21.15.103:0/1818929647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c40132030 0x7f7c40134450 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:25.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c47318700 1 -- 172.21.15.103:0/1818929647 shutdown_connections 2023-07-02T09:00:25.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c47318700 1 --2- 172.21.15.103:0/1818929647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c40132030 0x7f7c40134450 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:25.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c47318700 1 -- 172.21.15.103:0/1818929647 >> 172.21.15.103:0/1818929647 conn(0x7f7c4009d810 msgr2=0x7f7c4009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:25.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.507+0000 7f7c47318700 1 -- 172.21.15.103:0/1818929647 shutdown_connections 2023-07-02T09:00:25.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.508+0000 7f7c47318700 1 -- 172.21.15.103:0/1818929647 wait complete. 2023-07-02T09:00:25.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.508+0000 7f7c47318700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:25.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.509+0000 7f7c47318700 1 Processor -- start 2023-07-02T09:00:25.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.509+0000 7f7c47318700 1 -- start start 2023-07-02T09:00:25.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.509+0000 7f7c47318700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c4012f250 0x7f7c40117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:25.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.509+0000 7f7c47318700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c40128ef0 con 0x7f7c4012f250 2023-07-02T09:00:25.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.509+0000 7f7c450b4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c4012f250 0x7f7c40117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.509+0000 7f7c450b4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c4012f250 0x7f7c40117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44238/0 (socket says 172.21.15.103:44238) 2023-07-02T09:00:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.509+0000 7f7c450b4700 1 -- 172.21.15.103:0/2991570445 learned_addr learned my addr 172.21.15.103:0/2991570445 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.510+0000 7f7c450b4700 1 -- 172.21.15.103:0/2991570445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4011b6c0 con 0x7f7c4012f250 2023-07-02T09:00:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.510+0000 7f7c450b4700 1 --2- 172.21.15.103:0/2991570445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c4012f250 0x7f7c40117360 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.510+0000 7f7c35ffb700 1 -- 172.21.15.103:0/2991570445 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7c30005a60 con 0x7f7c4012f250 2023-07-02T09:00:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.510+0000 7f7c35ffb700 1 -- 172.21.15.103:0/2991570445 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7c30005bc0 con 0x7f7c4012f250 2023-07-02T09:00:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.510+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c40118f30 con 0x7f7c4012f250 2023-07-02T09:00:25.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.510+0000 7f7c35ffb700 1 -- 172.21.15.103:0/2991570445 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7c30005e60 con 0x7f7c4012f250 2023-07-02T09:00:25.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.510+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c40119810 con 0x7f7c4012f250 2023-07-02T09:00:25.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.511+0000 7f7c35ffb700 1 -- 172.21.15.103:0/2991570445 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7c300070b0 con 0x7f7c4012f250 2023-07-02T09:00:25.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.511+0000 7f7c35ffb700 1 --2- 172.21.15.103:0/2991570445 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7c240377c0 0x7f7c24039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:25.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.511+0000 7f7c35ffb700 1 -- 172.21.15.103:0/2991570445 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7c30046a90 con 0x7f7c4012f250 2023-07-02T09:00:25.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.511+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c40128cf0 con 0x7f7c4012f250 2023-07-02T09:00:25.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.511+0000 7f7c448b3700 1 --2- 172.21.15.103:0/2991570445 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7c240377c0 0x7f7c24039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:25.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.512+0000 7f7c448b3700 1 --2- 172.21.15.103:0/2991570445 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7c240377c0 0x7f7c24039cd0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:25.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.516+0000 7f7c35ffb700 1 -- 172.21.15.103:0/2991570445 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7c3000a640 con 0x7f7c4012f250 2023-07-02T09:00:25.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.707+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7c40061ea0 con 0x7f7c4012f250 2023-07-02T09:00:25.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.707+0000 7f7c35ffb700 1 -- 172.21.15.103:0/2991570445 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7c3000a640 con 0x7f7c4012f250 2023-07-02T09:00:25.708 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:25.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7c240377c0 msgr2=0x7f7c24039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:25.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 --2- 172.21.15.103:0/2991570445 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7c240377c0 0x7f7c24039cd0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:25.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c4012f250 msgr2=0x7f7c40117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:25.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 --2- 172.21.15.103:0/2991570445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c4012f250 0x7f7c40117360 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:25.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 shutdown_connections 2023-07-02T09:00:25.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 --2- 172.21.15.103:0/2991570445 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7c240377c0 0x7f7c24039cd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:25.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 --2- 172.21.15.103:0/2991570445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c4012f250 0x7f7c40117360 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:25.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 >> 172.21.15.103:0/2991570445 conn(0x7f7c4009d810 msgr2=0x7f7c4012e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:25.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 shutdown_connections 2023-07-02T09:00:25.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:25.710+0000 7f7c47318700 1 -- 172.21.15.103:0/2991570445 wait complete. 2023-07-02T09:00:25.723 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:26.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:26.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.864+0000 7f66482a0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:26.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.869+0000 7f66482a0700 1 Processor -- start 2023-07-02T09:00:26.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.869+0000 7f66482a0700 1 -- start start 2023-07-02T09:00:26.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f66482a0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6640133360 0x7f664009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:26.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f66482a0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664009e260 con 0x7f6640132f50 2023-07-02T09:00:26.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f66482a0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66400a1640 con 0x7f6640133360 2023-07-02T09:00:26.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f664583b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6640133360 0x7f664009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:26.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f664583b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6640133360 0x7f664009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44246/0 (socket says 172.21.15.103:44246) 2023-07-02T09:00:26.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f664583b700 1 -- 172.21.15.103:0/3533080967 learned_addr learned my addr 172.21.15.103:0/3533080967 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:26.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f664603c700 1 --1- 172.21.15.103:0/3533080967 >> v1:172.21.15.103:6789/0 conn(0x7f6640132f50 0x7f664009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47988/0 (socket says 172.21.15.103:47988) 2023-07-02T09:00:26.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f664583b700 1 -- 172.21.15.103:0/3533080967 >> v1:172.21.15.103:6789/0 conn(0x7f6640132f50 legacy=0x7f664009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:26.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.871+0000 7f664583b700 1 -- 172.21.15.103:0/3533080967 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640132a20 con 0x7f6640133360 2023-07-02T09:00:26.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.870+0000 7f664503a700 1 -- 172.21.15.103:0/3533080967 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467032816 0 0) 0x7f664009e260 con 0x7f6640132f50 2023-07-02T09:00:26.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.871+0000 7f664583b700 1 --2- 172.21.15.103:0/3533080967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640133360 0x7f664009cc90 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d393f6465fcead65 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:26.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.871+0000 7f664503a700 1 -- 172.21.15.103:0/3533080967 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6630009d90 con 0x7f6640133360 2023-07-02T09:00:26.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.871+0000 7f664503a700 1 -- 172.21.15.103:0/3533080967 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f663000e230 con 0x7f6640133360 2023-07-02T09:00:26.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.871+0000 7f664503a700 1 -- 172.21.15.103:0/3533080967 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f663000e4d0 con 0x7f6640133360 2023-07-02T09:00:26.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.871+0000 7f66482a0700 1 -- 172.21.15.103:0/3533080967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640133360 msgr2=0x7f664009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:26.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.871+0000 7f66482a0700 1 --2- 172.21.15.103:0/3533080967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640133360 0x7f664009cc90 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:26.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.872+0000 7f66482a0700 1 -- 172.21.15.103:0/3533080967 shutdown_connections 2023-07-02T09:00:26.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.872+0000 7f66482a0700 1 --2- 172.21.15.103:0/3533080967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640133360 0x7f664009cc90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:26.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.872+0000 7f66482a0700 1 -- 172.21.15.103:0/3533080967 >> 172.21.15.103:0/3533080967 conn(0x7f66400a2080 msgr2=0x7f66400a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:26.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.872+0000 7f66482a0700 1 -- 172.21.15.103:0/3533080967 shutdown_connections 2023-07-02T09:00:26.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.872+0000 7f66482a0700 1 -- 172.21.15.103:0/3533080967 wait complete. 2023-07-02T09:00:26.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.872+0000 7f66482a0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:26.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.873+0000 7f66482a0700 1 Processor -- start 2023-07-02T09:00:26.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.873+0000 7f66482a0700 1 -- start start 2023-07-02T09:00:26.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.873+0000 7f66482a0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640132f50 0x7f66401ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:26.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.873+0000 7f66482a0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664006d370 con 0x7f6640132f50 2023-07-02T09:00:26.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.873+0000 7f664603c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640132f50 0x7f66401ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:26.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f664603c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640132f50 0x7f66401ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44250/0 (socket says 172.21.15.103:44250) 2023-07-02T09:00:26.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f664603c700 1 -- 172.21.15.103:0/213709016 learned_addr learned my addr 172.21.15.103:0/213709016 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:26.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f664603c700 1 -- 172.21.15.103:0/213709016 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664006dde0 con 0x7f6640132f50 2023-07-02T09:00:26.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f664603c700 1 --2- 172.21.15.103:0/213709016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640132f50 0x7f66401ba9b0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:26.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f6636ffd700 1 -- 172.21.15.103:0/213709016 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f663c004b80 con 0x7f6640132f50 2023-07-02T09:00:26.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f6636ffd700 1 -- 172.21.15.103:0/213709016 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f663c004ce0 con 0x7f6640132f50 2023-07-02T09:00:26.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66401ba4f0 con 0x7f6640132f50 2023-07-02T09:00:26.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f6636ffd700 1 -- 172.21.15.103:0/213709016 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f663c00c4f0 con 0x7f6640132f50 2023-07-02T09:00:26.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.874+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6640071e40 con 0x7f6640132f50 2023-07-02T09:00:26.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.875+0000 7f6636ffd700 1 -- 172.21.15.103:0/213709016 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f663c00c650 con 0x7f6640132f50 2023-07-02T09:00:26.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.875+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6640061ea0 con 0x7f6640132f50 2023-07-02T09:00:26.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.875+0000 7f6636ffd700 1 --2- 172.21.15.103:0/213709016 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6624037810 0x7f6624039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:26.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.875+0000 7f6636ffd700 1 -- 172.21.15.103:0/213709016 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f663c046a30 con 0x7f6640132f50 2023-07-02T09:00:26.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.875+0000 7f664583b700 1 --2- 172.21.15.103:0/213709016 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6624037810 0x7f6624039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:26.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.876+0000 7f664583b700 1 --2- 172.21.15.103:0/213709016 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6624037810 0x7f6624039d20 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:26.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:26.880+0000 7f6636ffd700 1 -- 172.21.15.103:0/213709016 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f663c009c80 con 0x7f6640132f50 2023-07-02T09:00:27.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.071+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f66400725b0 con 0x7f6640132f50 2023-07-02T09:00:27.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.071+0000 7f6636ffd700 1 -- 172.21.15.103:0/213709016 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f663c0098a0 con 0x7f6640132f50 2023-07-02T09:00:27.072 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.073+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6624037810 msgr2=0x7f6624039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.073+0000 7f66482a0700 1 --2- 172.21.15.103:0/213709016 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6624037810 0x7f6624039d20 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.073+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640132f50 msgr2=0x7f66401ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.073+0000 7f66482a0700 1 --2- 172.21.15.103:0/213709016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640132f50 0x7f66401ba9b0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.073+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 shutdown_connections 2023-07-02T09:00:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.073+0000 7f66482a0700 1 --2- 172.21.15.103:0/213709016 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6624037810 0x7f6624039d20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.073+0000 7f66482a0700 1 --2- 172.21.15.103:0/213709016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6640132f50 0x7f66401ba9b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.073+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 >> 172.21.15.103:0/213709016 conn(0x7f66400a2080 msgr2=0x7f66400a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.074+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 shutdown_connections 2023-07-02T09:00:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:27.074+0000 7f66482a0700 1 -- 172.21.15.103:0/213709016 wait complete. 2023-07-02T09:00:27.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:28.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:28.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.221+0000 7f6d74f70700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:28.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.226+0000 7f6d74f70700 1 Processor -- start 2023-07-02T09:00:28.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.226+0000 7f6d74f70700 1 -- start start 2023-07-02T09:00:28.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d74f70700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6d70133360 0x7f6d7009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:28.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d74f70700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7009e260 con 0x7f6d70132f50 2023-07-02T09:00:28.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d74f70700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d700a1640 con 0x7f6d70133360 2023-07-02T09:00:28.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6dd9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6d70133360 0x7f6d7009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:28.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6e59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6d70132f50 0x7f6d7009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47996/0 (socket says 172.21.15.103:47996) 2023-07-02T09:00:28.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6dd9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6d70133360 0x7f6d7009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44260/0 (socket says 172.21.15.103:44260) 2023-07-02T09:00:28.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6e59c700 1 -- 172.21.15.103:0/3817308174 learned_addr learned my addr 172.21.15.103:0/3817308174 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:28.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6dd9b700 1 -- 172.21.15.103:0/3817308174 >> v1:172.21.15.103:6789/0 conn(0x7f6d70132f50 legacy=0x7f6d7009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:28.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6dd9b700 1 -- 172.21.15.103:0/3817308174 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d70132a20 con 0x7f6d70133360 2023-07-02T09:00:28.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6d59a700 1 -- 172.21.15.103:0/3817308174 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746961579 0 0) 0x7f6d7009e260 con 0x7f6d70132f50 2023-07-02T09:00:28.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6dd9b700 1 --2- 172.21.15.103:0/3817308174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70133360 0x7f6d7009cc90 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7526731b0a05d737 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:28.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6d59a700 1 -- 172.21.15.103:0/3817308174 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6d60009cd0 con 0x7f6d70133360 2023-07-02T09:00:28.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6d59a700 1 -- 172.21.15.103:0/3817308174 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6d60009e30 con 0x7f6d70133360 2023-07-02T09:00:28.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.227+0000 7f6d6d59a700 1 -- 172.21.15.103:0/3817308174 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6d6000e360 con 0x7f6d70133360 2023-07-02T09:00:28.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.228+0000 7f6d74f70700 1 -- 172.21.15.103:0/3817308174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70133360 msgr2=0x7f6d7009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:28.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.228+0000 7f6d74f70700 1 --2- 172.21.15.103:0/3817308174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70133360 0x7f6d7009cc90 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:28.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.228+0000 7f6d74f70700 1 -- 172.21.15.103:0/3817308174 shutdown_connections 2023-07-02T09:00:28.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.228+0000 7f6d74f70700 1 --2- 172.21.15.103:0/3817308174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70133360 0x7f6d7009cc90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:28.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.228+0000 7f6d74f70700 1 -- 172.21.15.103:0/3817308174 >> 172.21.15.103:0/3817308174 conn(0x7f6d700a2080 msgr2=0x7f6d700a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:28.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.228+0000 7f6d74f70700 1 -- 172.21.15.103:0/3817308174 shutdown_connections 2023-07-02T09:00:28.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.228+0000 7f6d74f70700 1 -- 172.21.15.103:0/3817308174 wait complete. 2023-07-02T09:00:28.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.228+0000 7f6d74f70700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:28.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.229+0000 7f6d74f70700 1 Processor -- start 2023-07-02T09:00:28.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.229+0000 7f6d74f70700 1 -- start start 2023-07-02T09:00:28.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.230+0000 7f6d74f70700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70132f50 0x7f6d701ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:28.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.230+0000 7f6d74f70700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7006d370 con 0x7f6d70132f50 2023-07-02T09:00:28.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.230+0000 7f6d6e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70132f50 0x7f6d701ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:28.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.230+0000 7f6d6e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70132f50 0x7f6d701ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44274/0 (socket says 172.21.15.103:44274) 2023-07-02T09:00:28.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.230+0000 7f6d6e59c700 1 -- 172.21.15.103:0/856683051 learned_addr learned my addr 172.21.15.103:0/856683051 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:28.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.230+0000 7f6d6e59c700 1 -- 172.21.15.103:0/856683051 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7006dde0 con 0x7f6d70132f50 2023-07-02T09:00:28.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.231+0000 7f6d6e59c700 1 --2- 172.21.15.103:0/856683051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70132f50 0x7f6d701ba9b0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:28.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.231+0000 7f6d5effd700 1 -- 172.21.15.103:0/856683051 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6d64004b80 con 0x7f6d70132f50 2023-07-02T09:00:28.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.231+0000 7f6d5effd700 1 -- 172.21.15.103:0/856683051 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6d64004ce0 con 0x7f6d70132f50 2023-07-02T09:00:28.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.231+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d701ba4f0 con 0x7f6d70132f50 2023-07-02T09:00:28.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.231+0000 7f6d5effd700 1 -- 172.21.15.103:0/856683051 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6d6400c4f0 con 0x7f6d70132f50 2023-07-02T09:00:28.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.231+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d70071e40 con 0x7f6d70132f50 2023-07-02T09:00:28.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.232+0000 7f6d5effd700 1 -- 172.21.15.103:0/856683051 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6d6400c650 con 0x7f6d70132f50 2023-07-02T09:00:28.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.232+0000 7f6d5effd700 1 --2- 172.21.15.103:0/856683051 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d500377c0 0x7f6d50039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:28.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.232+0000 7f6d5effd700 1 -- 172.21.15.103:0/856683051 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6d64046970 con 0x7f6d70132f50 2023-07-02T09:00:28.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.232+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d70061ea0 con 0x7f6d70132f50 2023-07-02T09:00:28.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.232+0000 7f6d6dd9b700 1 --2- 172.21.15.103:0/856683051 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d500377c0 0x7f6d50039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:28.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.233+0000 7f6d6dd9b700 1 --2- 172.21.15.103:0/856683051 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d500377c0 0x7f6d50039cd0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:28.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.237+0000 7f6d5effd700 1 -- 172.21.15.103:0/856683051 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6d64009c80 con 0x7f6d70132f50 2023-07-02T09:00:28.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.430+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6d700725b0 con 0x7f6d70132f50 2023-07-02T09:00:28.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.430+0000 7f6d5effd700 1 -- 172.21.15.103:0/856683051 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6d6402c020 con 0x7f6d70132f50 2023-07-02T09:00:28.431 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:28.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d500377c0 msgr2=0x7f6d50039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:28.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 --2- 172.21.15.103:0/856683051 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d500377c0 0x7f6d50039cd0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:28.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70132f50 msgr2=0x7f6d701ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:28.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 --2- 172.21.15.103:0/856683051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70132f50 0x7f6d701ba9b0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:28.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 shutdown_connections 2023-07-02T09:00:28.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 --2- 172.21.15.103:0/856683051 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d500377c0 0x7f6d50039cd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:28.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 --2- 172.21.15.103:0/856683051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d70132f50 0x7f6d701ba9b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:28.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 >> 172.21.15.103:0/856683051 conn(0x7f6d700a2080 msgr2=0x7f6d700a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:28.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 shutdown_connections 2023-07-02T09:00:28.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:28.432+0000 7f6d74f70700 1 -- 172.21.15.103:0/856683051 wait complete. 2023-07-02T09:00:28.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:29.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:29.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.587+0000 7f221b000700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:29.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.592+0000 7f221b000700 1 Processor -- start 2023-07-02T09:00:29.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.592+0000 7f221b000700 1 -- start start 2023-07-02T09:00:29.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f221b000700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2214133360 0x7f221409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:29.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f221b000700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221409e260 con 0x7f2214132f50 2023-07-02T09:00:29.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f221b000700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22140a1640 con 0x7f2214133360 2023-07-02T09:00:29.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f2213fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2214133360 0x7f221409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:29.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f2218d9c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2214132f50 0x7f221409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48002/0 (socket says 172.21.15.103:48002) 2023-07-02T09:00:29.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f2218d9c700 1 -- 172.21.15.103:0/3521938117 learned_addr learned my addr 172.21.15.103:0/3521938117 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:29.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f2213fff700 1 -- 172.21.15.103:0/3521938117 >> v1:172.21.15.103:6789/0 conn(0x7f2214132f50 legacy=0x7f221409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:29.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f2213fff700 1 -- 172.21.15.103:0/3521938117 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2214132a20 con 0x7f2214133360 2023-07-02T09:00:29.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.593+0000 7f2213fff700 1 --2- 172.21.15.103:0/3521938117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214133360 0x7f221409cc90 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3556de08e2171a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:29.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f22137fe700 1 -- 172.21.15.103:0/3521938117 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2200009cd0 con 0x7f2214133360 2023-07-02T09:00:29.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f22137fe700 1 -- 172.21.15.103:0/3521938117 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2200009e30 con 0x7f2214133360 2023-07-02T09:00:29.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f22137fe700 1 -- 172.21.15.103:0/3521938117 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f220000e360 con 0x7f2214133360 2023-07-02T09:00:29.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f221b000700 1 -- 172.21.15.103:0/3521938117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214133360 msgr2=0x7f221409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:29.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f221b000700 1 --2- 172.21.15.103:0/3521938117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214133360 0x7f221409cc90 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:29.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f221b000700 1 -- 172.21.15.103:0/3521938117 shutdown_connections 2023-07-02T09:00:29.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f221b000700 1 --2- 172.21.15.103:0/3521938117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214133360 0x7f221409cc90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:29.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f221b000700 1 -- 172.21.15.103:0/3521938117 >> 172.21.15.103:0/3521938117 conn(0x7f22140a2080 msgr2=0x7f22140a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:29.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f221b000700 1 -- 172.21.15.103:0/3521938117 shutdown_connections 2023-07-02T09:00:29.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.594+0000 7f221b000700 1 -- 172.21.15.103:0/3521938117 wait complete. 2023-07-02T09:00:29.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.595+0000 7f221b000700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:29.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.596+0000 7f221b000700 1 Processor -- start 2023-07-02T09:00:29.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.596+0000 7f221b000700 1 -- start start 2023-07-02T09:00:29.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.596+0000 7f221b000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214132f50 0x7f221412bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:29.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.596+0000 7f221b000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221406d260 con 0x7f2214132f50 2023-07-02T09:00:29.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f2218d9c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214132f50 0x7f221412bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:29.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f2218d9c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214132f50 0x7f221412bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44292/0 (socket says 172.21.15.103:44292) 2023-07-02T09:00:29.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f2218d9c700 1 -- 172.21.15.103:0/194777655 learned_addr learned my addr 172.21.15.103:0/194777655 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:29.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f2218d9c700 1 -- 172.21.15.103:0/194777655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221412bb90 con 0x7f2214132f50 2023-07-02T09:00:29.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f2218d9c700 1 --2- 172.21.15.103:0/194777655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214132f50 0x7f221412bea0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:29.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f22117fa700 1 -- 172.21.15.103:0/194777655 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2208005a60 con 0x7f2214132f50 2023-07-02T09:00:29.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f22117fa700 1 -- 172.21.15.103:0/194777655 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2208005bc0 con 0x7f2214132f50 2023-07-02T09:00:29.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221406f730 con 0x7f2214132f50 2023-07-02T09:00:29.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f22117fa700 1 -- 172.21.15.103:0/194777655 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2208005e60 con 0x7f2214132f50 2023-07-02T09:00:29.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.597+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221406f490 con 0x7f2214132f50 2023-07-02T09:00:29.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.598+0000 7f22117fa700 1 -- 172.21.15.103:0/194777655 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f22080070b0 con 0x7f2214132f50 2023-07-02T09:00:29.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.598+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2214061ea0 con 0x7f2214132f50 2023-07-02T09:00:29.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.598+0000 7f22117fa700 1 --2- 172.21.15.103:0/194777655 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21f80377c0 0x7f21f8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:29.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.598+0000 7f22117fa700 1 -- 172.21.15.103:0/194777655 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2208046a70 con 0x7f2214132f50 2023-07-02T09:00:29.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.599+0000 7f2213fff700 1 --2- 172.21.15.103:0/194777655 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21f80377c0 0x7f21f8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:29.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.599+0000 7f2213fff700 1 --2- 172.21.15.103:0/194777655 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21f80377c0 0x7f21f8039cd0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:29.603 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.603+0000 7f22117fa700 1 -- 172.21.15.103:0/194777655 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f220800a4a0 con 0x7f2214132f50 2023-07-02T09:00:29.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.795+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f221406f490 con 0x7f2214132f50 2023-07-02T09:00:29.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.795+0000 7f22117fa700 1 -- 172.21.15.103:0/194777655 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f221406f490 con 0x7f2214132f50 2023-07-02T09:00:29.796 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:29.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21f80377c0 msgr2=0x7f21f8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:29.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 --2- 172.21.15.103:0/194777655 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21f80377c0 0x7f21f8039cd0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:29.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214132f50 msgr2=0x7f221412bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:29.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 --2- 172.21.15.103:0/194777655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214132f50 0x7f221412bea0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:29.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 shutdown_connections 2023-07-02T09:00:29.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 --2- 172.21.15.103:0/194777655 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f21f80377c0 0x7f21f8039cd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:29.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 --2- 172.21.15.103:0/194777655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2214132f50 0x7f221412bea0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:29.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 >> 172.21.15.103:0/194777655 conn(0x7f22140a2080 msgr2=0x7f22141bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:29.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.797+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 shutdown_connections 2023-07-02T09:00:29.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:29.798+0000 7f221b000700 1 -- 172.21.15.103:0/194777655 wait complete. 2023-07-02T09:00:29.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:29.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~44s 2023-07-02T09:00:30.812 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:30.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.952+0000 7feb4621d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:30.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.957+0000 7feb4621d700 1 Processor -- start 2023-07-02T09:00:30.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.957+0000 7feb4621d700 1 -- start start 2023-07-02T09:00:30.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb4621d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb40133360 0x7feb4009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:30.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb4621d700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4009e260 con 0x7feb40132f50 2023-07-02T09:00:30.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb4621d700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb400a1640 con 0x7feb40133360 2023-07-02T09:00:30.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb3effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb40133360 0x7feb4009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:30.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb3effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb40133360 0x7feb4009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44306/0 (socket says 172.21.15.103:44306) 2023-07-02T09:00:30.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb3effd700 1 -- 172.21.15.103:0/3391333457 learned_addr learned my addr 172.21.15.103:0/3391333457 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:30.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb3f7fe700 1 --1- 172.21.15.103:0/3391333457 >> v1:172.21.15.103:6789/0 conn(0x7feb40132f50 0x7feb4009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48008/0 (socket says 172.21.15.103:48008) 2023-07-02T09:00:30.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb3effd700 1 -- 172.21.15.103:0/3391333457 >> v1:172.21.15.103:6789/0 conn(0x7feb40132f50 legacy=0x7feb4009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:30.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb3effd700 1 -- 172.21.15.103:0/3391333457 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40132a20 con 0x7feb40133360 2023-07-02T09:00:30.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb3e7fc700 1 -- 172.21.15.103:0/3391333457 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183775616 0 0) 0x7feb4009e260 con 0x7feb40132f50 2023-07-02T09:00:30.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.958+0000 7feb3effd700 1 --2- 172.21.15.103:0/3391333457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40133360 0x7feb4009cc90 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96770f2e6b3194d3 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:30.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb3e7fc700 1 -- 172.21.15.103:0/3391333457 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7feb30009d90 con 0x7feb40133360 2023-07-02T09:00:30.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb3e7fc700 1 -- 172.21.15.103:0/3391333457 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7feb3000e230 con 0x7feb40133360 2023-07-02T09:00:30.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb3e7fc700 1 -- 172.21.15.103:0/3391333457 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7feb3000e4d0 con 0x7feb40133360 2023-07-02T09:00:30.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb4621d700 1 -- 172.21.15.103:0/3391333457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40133360 msgr2=0x7feb4009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:30.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb4621d700 1 --2- 172.21.15.103:0/3391333457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40133360 0x7feb4009cc90 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:30.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb4621d700 1 -- 172.21.15.103:0/3391333457 shutdown_connections 2023-07-02T09:00:30.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb4621d700 1 --2- 172.21.15.103:0/3391333457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40133360 0x7feb4009cc90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:30.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb4621d700 1 -- 172.21.15.103:0/3391333457 >> 172.21.15.103:0/3391333457 conn(0x7feb400a2080 msgr2=0x7feb400a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:30.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb4621d700 1 -- 172.21.15.103:0/3391333457 shutdown_connections 2023-07-02T09:00:30.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.959+0000 7feb4621d700 1 -- 172.21.15.103:0/3391333457 wait complete. 2023-07-02T09:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.960+0000 7feb4621d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.961+0000 7feb4621d700 1 Processor -- start 2023-07-02T09:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.961+0000 7feb4621d700 1 -- start start 2023-07-02T09:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.961+0000 7feb4621d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40132f50 0x7feb4012c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.961+0000 7feb4621d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4006d2f0 con 0x7feb40132f50 2023-07-02T09:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.961+0000 7feb3f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40132f50 0x7feb4012c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb3f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40132f50 0x7feb4012c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44320/0 (socket says 172.21.15.103:44320) 2023-07-02T09:00:30.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb3f7fe700 1 -- 172.21.15.103:0/2486879723 learned_addr learned my addr 172.21.15.103:0/2486879723 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:30.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb3f7fe700 1 -- 172.21.15.103:0/2486879723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4006dd60 con 0x7feb40132f50 2023-07-02T09:00:30.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb3f7fe700 1 --2- 172.21.15.103:0/2486879723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40132f50 0x7feb4012c1a0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:30.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb27fff700 1 -- 172.21.15.103:0/2486879723 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7feb34004b80 con 0x7feb40132f50 2023-07-02T09:00:30.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb27fff700 1 -- 172.21.15.103:0/2486879723 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7feb34004ce0 con 0x7feb40132f50 2023-07-02T09:00:30.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb27fff700 1 -- 172.21.15.103:0/2486879723 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7feb3400c4f0 con 0x7feb40132f50 2023-07-02T09:00:30.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4012bc50 con 0x7feb40132f50 2023-07-02T09:00:30.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.962+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4006f850 con 0x7feb40132f50 2023-07-02T09:00:30.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.963+0000 7feb27fff700 1 -- 172.21.15.103:0/2486879723 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7feb3400c650 con 0x7feb40132f50 2023-07-02T09:00:30.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.963+0000 7feb27fff700 1 --2- 172.21.15.103:0/2486879723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb2c0377c0 0x7feb2c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:30.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.963+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4006dbb0 con 0x7feb40132f50 2023-07-02T09:00:30.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.963+0000 7feb27fff700 1 -- 172.21.15.103:0/2486879723 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7feb34046970 con 0x7feb40132f50 2023-07-02T09:00:30.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.963+0000 7feb3effd700 1 --2- 172.21.15.103:0/2486879723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb2c0377c0 0x7feb2c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:30.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.964+0000 7feb3effd700 1 --2- 172.21.15.103:0/2486879723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb2c0377c0 0x7feb2c039cd0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:30.968+0000 7feb27fff700 1 -- 172.21.15.103:0/2486879723 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7feb3400e1c0 con 0x7feb40132f50 2023-07-02T09:00:31.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.158+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7feb40061ea0 con 0x7feb40132f50 2023-07-02T09:00:31.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.158+0000 7feb27fff700 1 -- 172.21.15.103:0/2486879723 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7feb3401d5b0 con 0x7feb40132f50 2023-07-02T09:00:31.159 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:31.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.160+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb2c0377c0 msgr2=0x7feb2c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:31.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.160+0000 7feb4621d700 1 --2- 172.21.15.103:0/2486879723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb2c0377c0 0x7feb2c039cd0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:31.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.160+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40132f50 msgr2=0x7feb4012c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:31.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.160+0000 7feb4621d700 1 --2- 172.21.15.103:0/2486879723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40132f50 0x7feb4012c1a0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:31.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.160+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 shutdown_connections 2023-07-02T09:00:31.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.160+0000 7feb4621d700 1 --2- 172.21.15.103:0/2486879723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb2c0377c0 0x7feb2c039cd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:31.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.160+0000 7feb4621d700 1 --2- 172.21.15.103:0/2486879723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40132f50 0x7feb4012c1a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:31.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.161+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 >> 172.21.15.103:0/2486879723 conn(0x7feb400a2080 msgr2=0x7feb40130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:31.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.161+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 shutdown_connections 2023-07-02T09:00:31.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:31.161+0000 7feb4621d700 1 -- 172.21.15.103:0/2486879723 wait complete. 2023-07-02T09:00:31.174 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:32.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:32.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.314+0000 7efda346d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:32.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.319+0000 7efda346d700 1 Processor -- start 2023-07-02T09:00:32.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.319+0000 7efda346d700 1 -- start start 2023-07-02T09:00:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda346d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efd9c09a220 0x7efd9c1348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda346d700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c09cea0 con 0x7efd9c06d6e0 2023-07-02T09:00:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda346d700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c06d490 con 0x7efd9c09a220 2023-07-02T09:00:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda0a08700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efd9c09a220 0x7efd9c1348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda1209700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7efd9c06d6e0 0x7efd9c099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48016/0 (socket says 172.21.15.103:48016) 2023-07-02T09:00:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda0a08700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efd9c09a220 0x7efd9c1348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44336/0 (socket says 172.21.15.103:44336) 2023-07-02T09:00:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda1209700 1 -- 172.21.15.103:0/3108875067 learned_addr learned my addr 172.21.15.103:0/3108875067 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda0a08700 1 -- 172.21.15.103:0/3108875067 >> v1:172.21.15.103:6789/0 conn(0x7efd9c06d6e0 legacy=0x7efd9c099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda0a08700 1 -- 172.21.15.103:0/3108875067 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c06dc80 con 0x7efd9c09a220 2023-07-02T09:00:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efda0a08700 1 --2- 172.21.15.103:0/3108875067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c09a220 0x7efd9c1348e0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6729e340455b4d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.320+0000 7efd93fff700 1 -- 172.21.15.103:0/3108875067 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efd98009cd0 con 0x7efd9c09a220 2023-07-02T09:00:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efd93fff700 1 -- 172.21.15.103:0/3108875067 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efd98009e30 con 0x7efd9c09a220 2023-07-02T09:00:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efd93fff700 1 -- 172.21.15.103:0/3108875067 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efd9800e360 con 0x7efd9c09a220 2023-07-02T09:00:32.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efda346d700 1 -- 172.21.15.103:0/3108875067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c09a220 msgr2=0x7efd9c1348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:32.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efda346d700 1 --2- 172.21.15.103:0/3108875067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c09a220 0x7efd9c1348e0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:32.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efda346d700 1 -- 172.21.15.103:0/3108875067 shutdown_connections 2023-07-02T09:00:32.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efda346d700 1 --2- 172.21.15.103:0/3108875067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c09a220 0x7efd9c1348e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:32.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efda346d700 1 -- 172.21.15.103:0/3108875067 >> 172.21.15.103:0/3108875067 conn(0x7efd9c09d810 msgr2=0x7efd9c09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:32.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efda346d700 1 -- 172.21.15.103:0/3108875067 shutdown_connections 2023-07-02T09:00:32.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.321+0000 7efda346d700 1 -- 172.21.15.103:0/3108875067 wait complete. 2023-07-02T09:00:32.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.322+0000 7efda346d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:32.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.322+0000 7efda346d700 1 Processor -- start 2023-07-02T09:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.322+0000 7efda346d700 1 -- start start 2023-07-02T09:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.323+0000 7efda346d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c06d6e0 0x7efd9c12e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.323+0000 7efda346d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c11e020 con 0x7efd9c06d6e0 2023-07-02T09:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.323+0000 7efda1209700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c06d6e0 0x7efd9c12e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.323+0000 7efda1209700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c06d6e0 0x7efd9c12e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44342/0 (socket says 172.21.15.103:44342) 2023-07-02T09:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.323+0000 7efda1209700 1 -- 172.21.15.103:0/3478489589 learned_addr learned my addr 172.21.15.103:0/3478489589 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.324+0000 7efda1209700 1 -- 172.21.15.103:0/3478489589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c121270 con 0x7efd9c06d6e0 2023-07-02T09:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.324+0000 7efda1209700 1 --2- 172.21.15.103:0/3478489589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c06d6e0 0x7efd9c12e810 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:32.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.324+0000 7efd91ffb700 1 -- 172.21.15.103:0/3478489589 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efd8c005a60 con 0x7efd9c06d6e0 2023-07-02T09:00:32.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.324+0000 7efd91ffb700 1 -- 172.21.15.103:0/3478489589 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efd8c005bc0 con 0x7efd9c06d6e0 2023-07-02T09:00:32.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.324+0000 7efd91ffb700 1 -- 172.21.15.103:0/3478489589 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efd8c005e60 con 0x7efd9c06d6e0 2023-07-02T09:00:32.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.324+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c11dc80 con 0x7efd9c06d6e0 2023-07-02T09:00:32.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.324+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c121860 con 0x7efd9c06d6e0 2023-07-02T09:00:32.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.325+0000 7efd91ffb700 1 -- 172.21.15.103:0/3478489589 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7efd8c00c7b0 con 0x7efd9c06d6e0 2023-07-02T09:00:32.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.325+0000 7efd91ffb700 1 --2- 172.21.15.103:0/3478489589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd800377c0 0x7efd80039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:32.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.325+0000 7efd91ffb700 1 -- 172.21.15.103:0/3478489589 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7efd8c046bf0 con 0x7efd9c06d6e0 2023-07-02T09:00:32.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.325+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9c061ea0 con 0x7efd9c06d6e0 2023-07-02T09:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.325+0000 7efda0a08700 1 --2- 172.21.15.103:0/3478489589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd800377c0 0x7efd80039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.326+0000 7efda0a08700 1 --2- 172.21.15.103:0/3478489589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd800377c0 0x7efd80039cd0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.330+0000 7efd91ffb700 1 -- 172.21.15.103:0/3478489589 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7efd8c00a4a0 con 0x7efd9c06d6e0 2023-07-02T09:00:32.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.521+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efd9c11de10 con 0x7efd9c06d6e0 2023-07-02T09:00:32.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.522+0000 7efd91ffb700 1 -- 172.21.15.103:0/3478489589 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7efd8c00a4a0 con 0x7efd9c06d6e0 2023-07-02T09:00:32.522 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd800377c0 msgr2=0x7efd80039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 --2- 172.21.15.103:0/3478489589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd800377c0 0x7efd80039cd0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c06d6e0 msgr2=0x7efd9c12e810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 --2- 172.21.15.103:0/3478489589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c06d6e0 0x7efd9c12e810 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 shutdown_connections 2023-07-02T09:00:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 --2- 172.21.15.103:0/3478489589 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efd800377c0 0x7efd80039cd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 --2- 172.21.15.103:0/3478489589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd9c06d6e0 0x7efd9c12e810 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 >> 172.21.15.103:0/3478489589 conn(0x7efd9c09d810 msgr2=0x7efd9c117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 shutdown_connections 2023-07-02T09:00:32.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:32.524+0000 7efda346d700 1 -- 172.21.15.103:0/3478489589 wait complete. 2023-07-02T09:00:32.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:33.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:33.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.677+0000 7fa6baf90700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:33.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.682+0000 7fa6baf90700 1 Processor -- start 2023-07-02T09:00:33.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.683+0000 7fa6baf90700 1 -- start start 2023-07-02T09:00:33.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.683+0000 7fa6baf90700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa6b409a220 0x7fa6b41348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:33.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.683+0000 7fa6baf90700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b409cea0 con 0x7fa6b406d6e0 2023-07-02T09:00:33.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.683+0000 7fa6baf90700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b406d490 con 0x7fa6b409a220 2023-07-02T09:00:33.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.683+0000 7fa6b3fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa6b409a220 0x7fa6b41348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:33.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.683+0000 7fa6b8d2c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa6b406d6e0 0x7fa6b4099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41572/0 (socket says 172.21.15.103:41572) 2023-07-02T09:00:33.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.683+0000 7fa6b8d2c700 1 -- 172.21.15.103:0/3646989171 learned_addr learned my addr 172.21.15.103:0/3646989171 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:33.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.684+0000 7fa6b3fff700 1 -- 172.21.15.103:0/3646989171 >> v1:172.21.15.103:6789/0 conn(0x7fa6b406d6e0 legacy=0x7fa6b4099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:33.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.684+0000 7fa6b3fff700 1 -- 172.21.15.103:0/3646989171 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b406dc80 con 0x7fa6b409a220 2023-07-02T09:00:33.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.684+0000 7fa6b37fe700 1 -- 172.21.15.103:0/3646989171 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954427023 0 0) 0x7fa6b409cea0 con 0x7fa6b406d6e0 2023-07-02T09:00:33.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.684+0000 7fa6b3fff700 1 --2- 172.21.15.103:0/3646989171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b409a220 0x7fa6b41348e0 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141f75c06a59e170 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:33.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.684+0000 7fa6b37fe700 1 -- 172.21.15.103:0/3646989171 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa6a0010440 con 0x7fa6b409a220 2023-07-02T09:00:33.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.684+0000 7fa6b37fe700 1 -- 172.21.15.103:0/3646989171 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa6a00105a0 con 0x7fa6b409a220 2023-07-02T09:00:33.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.684+0000 7fa6b37fe700 1 -- 172.21.15.103:0/3646989171 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa6a0010840 con 0x7fa6b409a220 2023-07-02T09:00:33.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.685+0000 7fa6baf90700 1 -- 172.21.15.103:0/3646989171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b409a220 msgr2=0x7fa6b41348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:33.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.685+0000 7fa6baf90700 1 --2- 172.21.15.103:0/3646989171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b409a220 0x7fa6b41348e0 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:33.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.685+0000 7fa6baf90700 1 -- 172.21.15.103:0/3646989171 shutdown_connections 2023-07-02T09:00:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.685+0000 7fa6baf90700 1 --2- 172.21.15.103:0/3646989171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b409a220 0x7fa6b41348e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.685+0000 7fa6baf90700 1 -- 172.21.15.103:0/3646989171 >> 172.21.15.103:0/3646989171 conn(0x7fa6b409d810 msgr2=0x7fa6b409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.685+0000 7fa6baf90700 1 -- 172.21.15.103:0/3646989171 shutdown_connections 2023-07-02T09:00:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.685+0000 7fa6baf90700 1 -- 172.21.15.103:0/3646989171 wait complete. 2023-07-02T09:00:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.685+0000 7fa6baf90700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.686+0000 7fa6baf90700 1 Processor -- start 2023-07-02T09:00:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.686+0000 7fa6baf90700 1 -- start start 2023-07-02T09:00:33.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6baf90700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b406d6e0 0x7fa6b41c2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:33.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6baf90700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b412f020 con 0x7fa6b406d6e0 2023-07-02T09:00:33.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6b8d2c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b406d6e0 0x7fa6b41c2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:33.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6b8d2c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b406d6e0 0x7fa6b41c2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48196/0 (socket says 172.21.15.103:48196) 2023-07-02T09:00:33.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6b8d2c700 1 -- 172.21.15.103:0/3377628513 learned_addr learned my addr 172.21.15.103:0/3377628513 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:33.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6b8d2c700 1 -- 172.21.15.103:0/3377628513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b412f350 con 0x7fa6b406d6e0 2023-07-02T09:00:33.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6b8d2c700 1 --2- 172.21.15.103:0/3377628513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b406d6e0 0x7fa6b41c2ad0 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:33.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6b17fa700 1 -- 172.21.15.103:0/3377628513 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa6a8004b80 con 0x7fa6b406d6e0 2023-07-02T09:00:33.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6b17fa700 1 -- 172.21.15.103:0/3377628513 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa6a8004ce0 con 0x7fa6b406d6e0 2023-07-02T09:00:33.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b41b5670 con 0x7fa6b406d6e0 2023-07-02T09:00:33.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6b17fa700 1 -- 172.21.15.103:0/3377628513 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa6a800c550 con 0x7fa6b406d6e0 2023-07-02T09:00:33.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.687+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b4120870 con 0x7fa6b406d6e0 2023-07-02T09:00:33.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.688+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b412f1c0 con 0x7fa6b406d6e0 2023-07-02T09:00:33.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.688+0000 7fa6b17fa700 1 -- 172.21.15.103:0/3377628513 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa6a800c6b0 con 0x7fa6b406d6e0 2023-07-02T09:00:33.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.688+0000 7fa6b17fa700 1 --2- 172.21.15.103:0/3377628513 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6980377c0 0x7fa698039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:33.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.689+0000 7fa6b17fa700 1 -- 172.21.15.103:0/3377628513 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa6a80469d0 con 0x7fa6b406d6e0 2023-07-02T09:00:33.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.689+0000 7fa6b3fff700 1 --2- 172.21.15.103:0/3377628513 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6980377c0 0x7fa698039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:33.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.689+0000 7fa6b3fff700 1 --2- 172.21.15.103:0/3377628513 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6980377c0 0x7fa698039cd0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:33.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.693+0000 7fa6b17fa700 1 -- 172.21.15.103:0/3377628513 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa6a8009a80 con 0x7fa6b406d6e0 2023-07-02T09:00:33.894 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.894+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa6b4061ea0 con 0x7fa6b406d6e0 2023-07-02T09:00:33.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.894+0000 7fa6b17fa700 1 -- 172.21.15.103:0/3377628513 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa6a80098a0 con 0x7fa6b406d6e0 2023-07-02T09:00:33.895 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:33.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6980377c0 msgr2=0x7fa698039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:33.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 --2- 172.21.15.103:0/3377628513 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6980377c0 0x7fa698039cd0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:33.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b406d6e0 msgr2=0x7fa6b41c2ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:33.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 --2- 172.21.15.103:0/3377628513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b406d6e0 0x7fa6b41c2ad0 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:33.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 shutdown_connections 2023-07-02T09:00:33.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 --2- 172.21.15.103:0/3377628513 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6980377c0 0x7fa698039cd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:33.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 --2- 172.21.15.103:0/3377628513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6b406d6e0 0x7fa6b41c2ad0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:33.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 >> 172.21.15.103:0/3377628513 conn(0x7fa6b409d810 msgr2=0x7fa6b4070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:33.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 shutdown_connections 2023-07-02T09:00:33.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:33.897+0000 7fa6baf90700 1 -- 172.21.15.103:0/3377628513 wait complete. 2023-07-02T09:00:33.910 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:34.910 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:35.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.043+0000 7f3da216b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:35.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.048+0000 7f3da216b700 1 Processor -- start 2023-07-02T09:00:35.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.048+0000 7f3da216b700 1 -- start start 2023-07-02T09:00:35.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3da216b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3d9c132f50 0x7f3d9c133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:35.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3da216b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c09e260 con 0x7f3d9c09c580 2023-07-02T09:00:35.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3da216b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c0a1640 con 0x7f3d9c132f50 2023-07-02T09:00:35.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3d9b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3d9c132f50 0x7f3d9c133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:35.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3d9affd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3d9c09c580 0x7f3d9c09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41574/0 (socket says 172.21.15.103:41574) 2023-07-02T09:00:35.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3d9affd700 1 -- 172.21.15.103:0/1669461979 learned_addr learned my addr 172.21.15.103:0/1669461979 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:35.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3d9b7fe700 1 -- 172.21.15.103:0/1669461979 >> v1:172.21.15.103:6789/0 conn(0x7f3d9c09c580 legacy=0x7f3d9c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:35.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3d9b7fe700 1 -- 172.21.15.103:0/1669461979 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c132a20 con 0x7f3d9c132f50 2023-07-02T09:00:35.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.049+0000 7f3d9b7fe700 1 --2- 172.21.15.103:0/1669461979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c132f50 0x7f3d9c133360 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7337861bb7aed686 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:35.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3d9a7fc700 1 -- 172.21.15.103:0/1669461979 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3d8c009cd0 con 0x7f3d9c132f50 2023-07-02T09:00:35.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3d9a7fc700 1 -- 172.21.15.103:0/1669461979 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3d8c009e30 con 0x7f3d9c132f50 2023-07-02T09:00:35.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3d9a7fc700 1 -- 172.21.15.103:0/1669461979 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3d8c00e360 con 0x7f3d9c132f50 2023-07-02T09:00:35.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3da216b700 1 -- 172.21.15.103:0/1669461979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c132f50 msgr2=0x7f3d9c133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:35.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3da216b700 1 --2- 172.21.15.103:0/1669461979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c132f50 0x7f3d9c133360 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:35.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3da216b700 1 -- 172.21.15.103:0/1669461979 shutdown_connections 2023-07-02T09:00:35.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3da216b700 1 --2- 172.21.15.103:0/1669461979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c132f50 0x7f3d9c133360 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:35.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3da216b700 1 -- 172.21.15.103:0/1669461979 >> 172.21.15.103:0/1669461979 conn(0x7f3d9c0a2080 msgr2=0x7f3d9c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:35.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3da216b700 1 -- 172.21.15.103:0/1669461979 shutdown_connections 2023-07-02T09:00:35.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.050+0000 7f3da216b700 1 -- 172.21.15.103:0/1669461979 wait complete. 2023-07-02T09:00:35.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.051+0000 7f3da216b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:35.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.051+0000 7f3da216b700 1 Processor -- start 2023-07-02T09:00:35.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.052+0000 7f3da216b700 1 -- start start 2023-07-02T09:00:35.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.052+0000 7f3da216b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c09c580 0x7f3d9c12b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:35.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.052+0000 7f3da216b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c06f9d0 con 0x7f3d9c09c580 2023-07-02T09:00:35.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.052+0000 7f3d9b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c09c580 0x7f3d9c12b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:35.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3d9b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c09c580 0x7f3d9c12b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48202/0 (socket says 172.21.15.103:48202) 2023-07-02T09:00:35.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3d9b7fe700 1 -- 172.21.15.103:0/2192291 learned_addr learned my addr 172.21.15.103:0/2192291 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:35.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3d9b7fe700 1 -- 172.21.15.103:0/2192291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c133a00 con 0x7f3d9c09c580 2023-07-02T09:00:35.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3d9b7fe700 1 --2- 172.21.15.103:0/2192291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c09c580 0x7f3d9c12b7b0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:35.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3d93fff700 1 -- 172.21.15.103:0/2192291 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3d8c013530 con 0x7f3d9c09c580 2023-07-02T09:00:35.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c06d160 con 0x7f3d9c09c580 2023-07-02T09:00:35.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3d93fff700 1 -- 172.21.15.103:0/2192291 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3d8c013690 con 0x7f3d9c09c580 2023-07-02T09:00:35.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3d93fff700 1 -- 172.21.15.103:0/2192291 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3d8c013930 con 0x7f3d9c09c580 2023-07-02T09:00:35.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.053+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c06da70 con 0x7f3d9c09c580 2023-07-02T09:00:35.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.054+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c061ea0 con 0x7f3d9c09c580 2023-07-02T09:00:35.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.054+0000 7f3d93fff700 1 -- 172.21.15.103:0/2192291 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3d8c013b30 con 0x7f3d9c09c580 2023-07-02T09:00:35.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.054+0000 7f3d93fff700 1 --2- 172.21.15.103:0/2192291 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3d7c037770 0x7f3d7c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:35.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.054+0000 7f3d93fff700 1 -- 172.21.15.103:0/2192291 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3d8c018040 con 0x7f3d9c09c580 2023-07-02T09:00:35.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.055+0000 7f3d9affd700 1 --2- 172.21.15.103:0/2192291 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3d7c037770 0x7f3d7c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:35.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.055+0000 7f3d9affd700 1 --2- 172.21.15.103:0/2192291 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3d7c037770 0x7f3d7c039c80 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:35.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.059+0000 7f3d93fff700 1 -- 172.21.15.103:0/2192291 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3d8c018e00 con 0x7f3d9c09c580 2023-07-02T09:00:35.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.253+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3d9c0725b0 con 0x7f3d9c09c580 2023-07-02T09:00:35.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.253+0000 7f3d93fff700 1 -- 172.21.15.103:0/2192291 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3d8c010e60 con 0x7f3d9c09c580 2023-07-02T09:00:35.255 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:35.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3d7c037770 msgr2=0x7f3d7c039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:35.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 --2- 172.21.15.103:0/2192291 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3d7c037770 0x7f3d7c039c80 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:35.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c09c580 msgr2=0x7f3d9c12b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:35.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 --2- 172.21.15.103:0/2192291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c09c580 0x7f3d9c12b7b0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:35.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 shutdown_connections 2023-07-02T09:00:35.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 --2- 172.21.15.103:0/2192291 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3d7c037770 0x7f3d7c039c80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:35.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 --2- 172.21.15.103:0/2192291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d9c09c580 0x7f3d9c12b7b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:35.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 >> 172.21.15.103:0/2192291 conn(0x7f3d9c0a2080 msgr2=0x7f3d9c1babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:35.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 shutdown_connections 2023-07-02T09:00:35.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:35.256+0000 7f3da216b700 1 -- 172.21.15.103:0/2192291 wait complete. 2023-07-02T09:00:35.270 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:35.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~50s 2023-07-02T09:00:36.271 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:36.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.410+0000 7fde50681700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:36.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.415+0000 7fde50681700 1 Processor -- start 2023-07-02T09:00:36.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.415+0000 7fde50681700 1 -- start start 2023-07-02T09:00:36.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde50681700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fde4809a220 0x7fde481348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:36.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde50681700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4809cea0 con 0x7fde4806d6e0 2023-07-02T09:00:36.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde50681700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4806d490 con 0x7fde4809a220 2023-07-02T09:00:36.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4dc1c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fde4809a220 0x7fde481348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:36.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4e41d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fde4806d6e0 0x7fde48099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41584/0 (socket says 172.21.15.103:41584) 2023-07-02T09:00:36.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4e41d700 1 -- 172.21.15.103:0/3051271817 learned_addr learned my addr 172.21.15.103:0/3051271817 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:36.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4dc1c700 1 -- 172.21.15.103:0/3051271817 >> v1:172.21.15.103:6789/0 conn(0x7fde4806d6e0 legacy=0x7fde48099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:36.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4dc1c700 1 -- 172.21.15.103:0/3051271817 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4806dc80 con 0x7fde4809a220 2023-07-02T09:00:36.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4dc1c700 1 --2- 172.21.15.103:0/3051271817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4809a220 0x7fde481348e0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7e47533900ba98 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:36.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4d41b700 1 -- 172.21.15.103:0/3051271817 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fde44009cd0 con 0x7fde4809a220 2023-07-02T09:00:36.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4d41b700 1 -- 172.21.15.103:0/3051271817 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fde44009e30 con 0x7fde4809a220 2023-07-02T09:00:36.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.416+0000 7fde4d41b700 1 -- 172.21.15.103:0/3051271817 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fde4400e360 con 0x7fde4809a220 2023-07-02T09:00:36.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.417+0000 7fde50681700 1 -- 172.21.15.103:0/3051271817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4809a220 msgr2=0x7fde481348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:36.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.417+0000 7fde50681700 1 --2- 172.21.15.103:0/3051271817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4809a220 0x7fde481348e0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:36.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.417+0000 7fde50681700 1 -- 172.21.15.103:0/3051271817 shutdown_connections 2023-07-02T09:00:36.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.417+0000 7fde50681700 1 --2- 172.21.15.103:0/3051271817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4809a220 0x7fde481348e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:36.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.417+0000 7fde50681700 1 -- 172.21.15.103:0/3051271817 >> 172.21.15.103:0/3051271817 conn(0x7fde4809d810 msgr2=0x7fde4809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:36.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.417+0000 7fde50681700 1 -- 172.21.15.103:0/3051271817 shutdown_connections 2023-07-02T09:00:36.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.417+0000 7fde50681700 1 -- 172.21.15.103:0/3051271817 wait complete. 2023-07-02T09:00:36.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.417+0000 7fde50681700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:36.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.418+0000 7fde50681700 1 Processor -- start 2023-07-02T09:00:36.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.418+0000 7fde50681700 1 -- start start 2023-07-02T09:00:36.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.419+0000 7fde50681700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4806d6e0 0x7fde481c3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:36.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.419+0000 7fde50681700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4811de10 con 0x7fde4806d6e0 2023-07-02T09:00:36.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.419+0000 7fde4e41d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4806d6e0 0x7fde481c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:36.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.419+0000 7fde4e41d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4806d6e0 0x7fde481c3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48222/0 (socket says 172.21.15.103:48222) 2023-07-02T09:00:36.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.419+0000 7fde4e41d700 1 -- 172.21.15.103:0/165895092 learned_addr learned my addr 172.21.15.103:0/165895092 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:36.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.419+0000 7fde4e41d700 1 -- 172.21.15.103:0/165895092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4811f650 con 0x7fde4806d6e0 2023-07-02T09:00:36.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.419+0000 7fde4e41d700 1 --2- 172.21.15.103:0/165895092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4806d6e0 0x7fde481c3930 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:36.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.420+0000 7fde3effd700 1 -- 172.21.15.103:0/165895092 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fde38005a60 con 0x7fde4806d6e0 2023-07-02T09:00:36.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.420+0000 7fde3effd700 1 -- 172.21.15.103:0/165895092 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fde38005bc0 con 0x7fde4806d6e0 2023-07-02T09:00:36.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.420+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4811db00 con 0x7fde4806d6e0 2023-07-02T09:00:36.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.420+0000 7fde3effd700 1 -- 172.21.15.103:0/165895092 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fde38005e60 con 0x7fde4806d6e0 2023-07-02T09:00:36.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.420+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde48120f20 con 0x7fde4806d6e0 2023-07-02T09:00:36.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.421+0000 7fde3effd700 1 -- 172.21.15.103:0/165895092 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fde3800c7b0 con 0x7fde4806d6e0 2023-07-02T09:00:36.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.421+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde48061ea0 con 0x7fde4806d6e0 2023-07-02T09:00:36.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.421+0000 7fde3effd700 1 --2- 172.21.15.103:0/165895092 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fde2c0399d0 0x7fde2c03bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:36.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.421+0000 7fde3effd700 1 -- 172.21.15.103:0/165895092 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fde38046b60 con 0x7fde4806d6e0 2023-07-02T09:00:36.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.421+0000 7fde4dc1c700 1 --2- 172.21.15.103:0/165895092 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fde2c0399d0 0x7fde2c03bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:36.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.421+0000 7fde4dc1c700 1 --2- 172.21.15.103:0/165895092 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fde2c0399d0 0x7fde2c03bee0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:36.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.426+0000 7fde3effd700 1 -- 172.21.15.103:0/165895092 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fde3800a4a0 con 0x7fde4806d6e0 2023-07-02T09:00:36.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.615+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fde48073ca0 con 0x7fde4806d6e0 2023-07-02T09:00:36.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.616+0000 7fde3effd700 1 -- 172.21.15.103:0/165895092 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fde3800a4a0 con 0x7fde4806d6e0 2023-07-02T09:00:36.617 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:36.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fde2c0399d0 msgr2=0x7fde2c03bee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:36.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 --2- 172.21.15.103:0/165895092 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fde2c0399d0 0x7fde2c03bee0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:36.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4806d6e0 msgr2=0x7fde481c3930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:36.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 --2- 172.21.15.103:0/165895092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4806d6e0 0x7fde481c3930 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:36.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 shutdown_connections 2023-07-02T09:00:36.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 --2- 172.21.15.103:0/165895092 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fde2c0399d0 0x7fde2c03bee0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:36.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 --2- 172.21.15.103:0/165895092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4806d6e0 0x7fde481c3930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:36.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 >> 172.21.15.103:0/165895092 conn(0x7fde4809d810 msgr2=0x7fde48117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:36.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 shutdown_connections 2023-07-02T09:00:36.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:36.618+0000 7fde50681700 1 -- 172.21.15.103:0/165895092 wait complete. 2023-07-02T09:00:36.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:37.633 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:37.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.774+0000 7f1d9580a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:37.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.779+0000 7f1d9580a700 1 Processor -- start 2023-07-02T09:00:37.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.779+0000 7f1d9580a700 1 -- start start 2023-07-02T09:00:37.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d9580a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d90133360 0x7f1d9009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:37.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d9580a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9009e260 con 0x7f1d90132f50 2023-07-02T09:00:37.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d9580a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d900a1640 con 0x7f1d90133360 2023-07-02T09:00:37.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d8e7fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d90133360 0x7f1d9009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:37.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d8effd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1d90132f50 0x7f1d9009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41586/0 (socket says 172.21.15.103:41586) 2023-07-02T09:00:37.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d8effd700 1 -- 172.21.15.103:0/3560620078 learned_addr learned my addr 172.21.15.103:0/3560620078 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d8e7fc700 1 -- 172.21.15.103:0/3560620078 >> v1:172.21.15.103:6789/0 conn(0x7f1d90132f50 legacy=0x7f1d9009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d8e7fc700 1 -- 172.21.15.103:0/3560620078 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d90132a20 con 0x7f1d90133360 2023-07-02T09:00:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d8dffb700 1 -- 172.21.15.103:0/3560620078 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914616118 0 0) 0x7f1d9009e260 con 0x7f1d90132f50 2023-07-02T09:00:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.780+0000 7f1d8e7fc700 1 --2- 172.21.15.103:0/3560620078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90133360 0x7f1d9009cc90 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843dcb1bd8ce1ba3 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d8dffb700 1 -- 172.21.15.103:0/3560620078 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1d80009cd0 con 0x7f1d90133360 2023-07-02T09:00:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d8dffb700 1 -- 172.21.15.103:0/3560620078 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1d80009e30 con 0x7f1d90133360 2023-07-02T09:00:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d8dffb700 1 -- 172.21.15.103:0/3560620078 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1d8000e360 con 0x7f1d90133360 2023-07-02T09:00:37.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d9580a700 1 -- 172.21.15.103:0/3560620078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90133360 msgr2=0x7f1d9009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:37.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d9580a700 1 --2- 172.21.15.103:0/3560620078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90133360 0x7f1d9009cc90 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:37.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d9580a700 1 -- 172.21.15.103:0/3560620078 shutdown_connections 2023-07-02T09:00:37.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d9580a700 1 --2- 172.21.15.103:0/3560620078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90133360 0x7f1d9009cc90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:37.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d9580a700 1 -- 172.21.15.103:0/3560620078 >> 172.21.15.103:0/3560620078 conn(0x7f1d900a2080 msgr2=0x7f1d900a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d9580a700 1 -- 172.21.15.103:0/3560620078 shutdown_connections 2023-07-02T09:00:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.781+0000 7f1d9580a700 1 -- 172.21.15.103:0/3560620078 wait complete. 2023-07-02T09:00:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.782+0000 7f1d9580a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.782+0000 7f1d9580a700 1 Processor -- start 2023-07-02T09:00:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.783+0000 7f1d9580a700 1 -- start start 2023-07-02T09:00:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.783+0000 7f1d9580a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90132f50 0x7f1d901ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.783+0000 7f1d9580a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9006d370 con 0x7f1d90132f50 2023-07-02T09:00:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.783+0000 7f1d8effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90132f50 0x7f1d901ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:37.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d8effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90132f50 0x7f1d901ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48240/0 (socket says 172.21.15.103:48240) 2023-07-02T09:00:37.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d8effd700 1 -- 172.21.15.103:0/2508743211 learned_addr learned my addr 172.21.15.103:0/2508743211 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:37.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d8effd700 1 -- 172.21.15.103:0/2508743211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9006dde0 con 0x7f1d90132f50 2023-07-02T09:00:37.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d8effd700 1 --2- 172.21.15.103:0/2508743211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90132f50 0x7f1d901ba9b0 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:37.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d77fff700 1 -- 172.21.15.103:0/2508743211 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1d84004b80 con 0x7f1d90132f50 2023-07-02T09:00:37.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d77fff700 1 -- 172.21.15.103:0/2508743211 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1d84004ce0 con 0x7f1d90132f50 2023-07-02T09:00:37.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d901ba4f0 con 0x7f1d90132f50 2023-07-02T09:00:37.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d77fff700 1 -- 172.21.15.103:0/2508743211 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1d8400c4f0 con 0x7f1d90132f50 2023-07-02T09:00:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.784+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d90071e40 con 0x7f1d90132f50 2023-07-02T09:00:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.785+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d90061ea0 con 0x7f1d90132f50 2023-07-02T09:00:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.785+0000 7f1d77fff700 1 -- 172.21.15.103:0/2508743211 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f1d8400c650 con 0x7f1d90132f50 2023-07-02T09:00:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.785+0000 7f1d77fff700 1 --2- 172.21.15.103:0/2508743211 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1d7c0377c0 0x7f1d7c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.785+0000 7f1d77fff700 1 -- 172.21.15.103:0/2508743211 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f1d84046970 con 0x7f1d90132f50 2023-07-02T09:00:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.786+0000 7f1d8e7fc700 1 --2- 172.21.15.103:0/2508743211 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1d7c0377c0 0x7f1d7c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.786+0000 7f1d8e7fc700 1 --2- 172.21.15.103:0/2508743211 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1d7c0377c0 0x7f1d7c039cd0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:37.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.790+0000 7f1d77fff700 1 -- 172.21.15.103:0/2508743211 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f1d84009c80 con 0x7f1d90132f50 2023-07-02T09:00:37.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.984+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1d900725b0 con 0x7f1d90132f50 2023-07-02T09:00:37.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.984+0000 7f1d77fff700 1 -- 172.21.15.103:0/2508743211 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f1d840098a0 con 0x7f1d90132f50 2023-07-02T09:00:37.986 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.986+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1d7c0377c0 msgr2=0x7f1d7c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.986+0000 7f1d9580a700 1 --2- 172.21.15.103:0/2508743211 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1d7c0377c0 0x7f1d7c039cd0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.986+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90132f50 msgr2=0x7f1d901ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.986+0000 7f1d9580a700 1 --2- 172.21.15.103:0/2508743211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90132f50 0x7f1d901ba9b0 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.987+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 shutdown_connections 2023-07-02T09:00:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.987+0000 7f1d9580a700 1 --2- 172.21.15.103:0/2508743211 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1d7c0377c0 0x7f1d7c039cd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.987+0000 7f1d9580a700 1 --2- 172.21.15.103:0/2508743211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d90132f50 0x7f1d901ba9b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.987+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 >> 172.21.15.103:0/2508743211 conn(0x7f1d900a2080 msgr2=0x7f1d900a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.987+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 shutdown_connections 2023-07-02T09:00:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:37.987+0000 7f1d9580a700 1 -- 172.21.15.103:0/2508743211 wait complete. 2023-07-02T09:00:38.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:39.002 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:39.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.141+0000 7f865a05c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:39.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.146+0000 7f865a05c700 1 Processor -- start 2023-07-02T09:00:39.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.146+0000 7f865a05c700 1 -- start start 2023-07-02T09:00:39.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f865a05c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f865412f250 0x7f865412f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:39.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f865a05c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865406df60 con 0x7f8654098a00 2023-07-02T09:00:39.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f865a05c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865409a710 con 0x7f865412f250 2023-07-02T09:00:39.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f86537fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f865412f250 0x7f865412f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:39.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f8652ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f8654098a00 0x7f8654098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41598/0 (socket says 172.21.15.103:41598) 2023-07-02T09:00:39.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f8652ffd700 1 -- 172.21.15.103:0/1262466497 learned_addr learned my addr 172.21.15.103:0/1262466497 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f86537fe700 1 -- 172.21.15.103:0/1262466497 >> v1:172.21.15.103:6789/0 conn(0x7f8654098a00 legacy=0x7f8654098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f86527fc700 1 -- 172.21.15.103:0/1262466497 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767843713 0 0) 0x7f865406df60 con 0x7f8654098a00 2023-07-02T09:00:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f86537fe700 1 -- 172.21.15.103:0/1262466497 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865412ed20 con 0x7f865412f250 2023-07-02T09:00:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.147+0000 7f86537fe700 1 --2- 172.21.15.103:0/1262466497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865412f250 0x7f865412f660 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b02d6236266c02e server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f86527fc700 1 -- 172.21.15.103:0/1262466497 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f864400d450 con 0x7f865412f250 2023-07-02T09:00:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f86527fc700 1 -- 172.21.15.103:0/1262466497 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f864400d5b0 con 0x7f865412f250 2023-07-02T09:00:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f86527fc700 1 -- 172.21.15.103:0/1262466497 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f864400d850 con 0x7f865412f250 2023-07-02T09:00:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f865a05c700 1 -- 172.21.15.103:0/1262466497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865412f250 msgr2=0x7f865412f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f865a05c700 1 --2- 172.21.15.103:0/1262466497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865412f250 0x7f865412f660 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f865a05c700 1 -- 172.21.15.103:0/1262466497 shutdown_connections 2023-07-02T09:00:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f865a05c700 1 --2- 172.21.15.103:0/1262466497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f865412f250 0x7f865412f660 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f865a05c700 1 -- 172.21.15.103:0/1262466497 >> 172.21.15.103:0/1262466497 conn(0x7f865409d810 msgr2=0x7f865409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f865a05c700 1 -- 172.21.15.103:0/1262466497 shutdown_connections 2023-07-02T09:00:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.148+0000 7f865a05c700 1 -- 172.21.15.103:0/1262466497 wait complete. 2023-07-02T09:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.149+0000 7f865a05c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.150+0000 7f865a05c700 1 Processor -- start 2023-07-02T09:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.150+0000 7f865a05c700 1 -- start start 2023-07-02T09:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.150+0000 7f865a05c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8654098a00 0x7f865411b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.150+0000 7f865a05c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865411e820 con 0x7f8654098a00 2023-07-02T09:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f86537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8654098a00 0x7f865411b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f86537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8654098a00 0x7f865411b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48254/0 (socket says 172.21.15.103:48254) 2023-07-02T09:00:39.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f86537fe700 1 -- 172.21.15.103:0/1390807994 learned_addr learned my addr 172.21.15.103:0/1390807994 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f86537fe700 1 -- 172.21.15.103:0/1390807994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865411dba0 con 0x7f8654098a00 2023-07-02T09:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f86537fe700 1 --2- 172.21.15.103:0/1390807994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8654098a00 0x7f865411b770 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f864bfff700 1 -- 172.21.15.103:0/1390807994 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8644009040 con 0x7f8654098a00 2023-07-02T09:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f864bfff700 1 -- 172.21.15.103:0/1390807994 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f864400be50 con 0x7f8654098a00 2023-07-02T09:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f864bfff700 1 -- 172.21.15.103:0/1390807994 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f864400ca80 con 0x7f8654098a00 2023-07-02T09:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865411ea80 con 0x7f8654098a00 2023-07-02T09:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.151+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8654128fb0 con 0x7f8654098a00 2023-07-02T09:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.152+0000 7f864bfff700 1 -- 172.21.15.103:0/1390807994 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f864400cc30 con 0x7f8654098a00 2023-07-02T09:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.152+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8654061ea0 con 0x7f8654098a00 2023-07-02T09:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.152+0000 7f864bfff700 1 --2- 172.21.15.103:0/1390807994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f86340377b0 0x7f8634039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.152+0000 7f864bfff700 1 -- 172.21.15.103:0/1390807994 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f8644012040 con 0x7f8654098a00 2023-07-02T09:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.153+0000 7f8652ffd700 1 --2- 172.21.15.103:0/1390807994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f86340377b0 0x7f8634039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.153+0000 7f8652ffd700 1 --2- 172.21.15.103:0/1390807994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f86340377b0 0x7f8634039cc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:39.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.157+0000 7f864bfff700 1 -- 172.21.15.103:0/1390807994 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f864400ac70 con 0x7f8654098a00 2023-07-02T09:00:39.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.351+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8654073ca0 con 0x7f8654098a00 2023-07-02T09:00:39.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.351+0000 7f864bfff700 1 -- 172.21.15.103:0/1390807994 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f864400aea0 con 0x7f8654098a00 2023-07-02T09:00:39.353 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:39.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.353+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f86340377b0 msgr2=0x7f8634039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:39.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.353+0000 7f865a05c700 1 --2- 172.21.15.103:0/1390807994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f86340377b0 0x7f8634039cc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:39.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.353+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8654098a00 msgr2=0x7f865411b770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:39.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.353+0000 7f865a05c700 1 --2- 172.21.15.103:0/1390807994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8654098a00 0x7f865411b770 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:39.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.353+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 shutdown_connections 2023-07-02T09:00:39.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.353+0000 7f865a05c700 1 --2- 172.21.15.103:0/1390807994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f86340377b0 0x7f8634039cc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:39.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.353+0000 7f865a05c700 1 --2- 172.21.15.103:0/1390807994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8654098a00 0x7f865411b770 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:39.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.353+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 >> 172.21.15.103:0/1390807994 conn(0x7f865409d810 msgr2=0x7f86540a0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:39.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.354+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 shutdown_connections 2023-07-02T09:00:39.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:39.354+0000 7f865a05c700 1 -- 172.21.15.103:0/1390807994 wait complete. 2023-07-02T09:00:39.368 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:40.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:40.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.508+0000 7f3cbede0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:40.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.513+0000 7f3cbede0700 1 Processor -- start 2023-07-02T09:00:40.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.513+0000 7f3cbede0700 1 -- start start 2023-07-02T09:00:40.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cbede0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3cb8132030 0x7f3cb8134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:40.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cbede0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb806df60 con 0x7f3cb812f250 2023-07-02T09:00:40.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cbede0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb809a710 con 0x7f3cb8132030 2023-07-02T09:00:40.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cb7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3cb8132030 0x7f3cb8134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:40.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cbcb7c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3cb812f250 0x7f3cb812f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41606/0 (socket says 172.21.15.103:41606) 2023-07-02T09:00:40.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cb7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3cb8132030 0x7f3cb8134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48260/0 (socket says 172.21.15.103:48260) 2023-07-02T09:00:40.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cbcb7c700 1 -- 172.21.15.103:0/217621780 learned_addr learned my addr 172.21.15.103:0/217621780 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:40.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cb7fff700 1 -- 172.21.15.103:0/217621780 >> v1:172.21.15.103:6789/0 conn(0x7f3cb812f250 legacy=0x7f3cb812f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:40.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cb7fff700 1 -- 172.21.15.103:0/217621780 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb812ed20 con 0x7f3cb8132030 2023-07-02T09:00:40.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cb77fe700 1 -- 172.21.15.103:0/217621780 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792087710 0 0) 0x7f3cb806df60 con 0x7f3cb812f250 2023-07-02T09:00:40.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.514+0000 7f3cb7fff700 1 --2- 172.21.15.103:0/217621780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb8132030 0x7f3cb8134450 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7249f0935f7cc29d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:40.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cb77fe700 1 -- 172.21.15.103:0/217621780 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3ca4009cd0 con 0x7f3cb8132030 2023-07-02T09:00:40.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cb77fe700 1 -- 172.21.15.103:0/217621780 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3ca4009e30 con 0x7f3cb8132030 2023-07-02T09:00:40.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cb77fe700 1 -- 172.21.15.103:0/217621780 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3ca400e360 con 0x7f3cb8132030 2023-07-02T09:00:40.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cbede0700 1 -- 172.21.15.103:0/217621780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb8132030 msgr2=0x7f3cb8134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:40.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cbede0700 1 --2- 172.21.15.103:0/217621780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb8132030 0x7f3cb8134450 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:40.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cbede0700 1 -- 172.21.15.103:0/217621780 shutdown_connections 2023-07-02T09:00:40.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cbede0700 1 --2- 172.21.15.103:0/217621780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb8132030 0x7f3cb8134450 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:40.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cbede0700 1 -- 172.21.15.103:0/217621780 >> 172.21.15.103:0/217621780 conn(0x7f3cb809d810 msgr2=0x7f3cb809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:40.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cbede0700 1 -- 172.21.15.103:0/217621780 shutdown_connections 2023-07-02T09:00:40.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.515+0000 7f3cbede0700 1 -- 172.21.15.103:0/217621780 wait complete. 2023-07-02T09:00:40.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.516+0000 7f3cbede0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:40.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.516+0000 7f3cbede0700 1 Processor -- start 2023-07-02T09:00:40.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.517+0000 7f3cbede0700 1 -- start start 2023-07-02T09:00:40.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.517+0000 7f3cbede0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb812f250 0x7f3cb812b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:40.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.517+0000 7f3cbede0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb811a010 con 0x7f3cb812f250 2023-07-02T09:00:40.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.517+0000 7f3cbcb7c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb812f250 0x7f3cb812b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.517+0000 7f3cbcb7c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb812f250 0x7f3cb812b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48272/0 (socket says 172.21.15.103:48272) 2023-07-02T09:00:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.517+0000 7f3cbcb7c700 1 -- 172.21.15.103:0/4047556250 learned_addr learned my addr 172.21.15.103:0/4047556250 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.518+0000 7f3cbcb7c700 1 -- 172.21.15.103:0/4047556250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8070cf0 con 0x7f3cb812f250 2023-07-02T09:00:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.518+0000 7f3cbcb7c700 1 --2- 172.21.15.103:0/4047556250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb812f250 0x7f3cb812b5a0 crc :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.518+0000 7f3cb57fa700 1 -- 172.21.15.103:0/4047556250 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3cac004b80 con 0x7f3cb812f250 2023-07-02T09:00:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.518+0000 7f3cb57fa700 1 -- 172.21.15.103:0/4047556250 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3cac004ce0 con 0x7f3cb812f250 2023-07-02T09:00:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.518+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb8070f50 con 0x7f3cb812f250 2023-07-02T09:00:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.518+0000 7f3cb57fa700 1 -- 172.21.15.103:0/4047556250 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3cac00c4f0 con 0x7f3cb812f250 2023-07-02T09:00:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.518+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb8118720 con 0x7f3cb812f250 2023-07-02T09:00:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.519+0000 7f3cb57fa700 1 -- 172.21.15.103:0/4047556250 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3cac00c650 con 0x7f3cb812f250 2023-07-02T09:00:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.519+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb8071140 con 0x7f3cb812f250 2023-07-02T09:00:40.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.519+0000 7f3cb57fa700 1 --2- 172.21.15.103:0/4047556250 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c9c0377c0 0x7f3c9c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:40.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.519+0000 7f3cb57fa700 1 -- 172.21.15.103:0/4047556250 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3cac046970 con 0x7f3cb812f250 2023-07-02T09:00:40.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.519+0000 7f3cb7fff700 1 --2- 172.21.15.103:0/4047556250 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c9c0377c0 0x7f3c9c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:40.527 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.520+0000 7f3cb7fff700 1 --2- 172.21.15.103:0/4047556250 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c9c0377c0 0x7f3c9c039cd0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:40.528 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.524+0000 7f3cb57fa700 1 -- 172.21.15.103:0/4047556250 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3cac009c80 con 0x7f3cb812f250 2023-07-02T09:00:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.716+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3cb8061ea0 con 0x7f3cb812f250 2023-07-02T09:00:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.717+0000 7f3cb57fa700 1 -- 172.21.15.103:0/4047556250 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3cac0098a0 con 0x7f3cb812f250 2023-07-02T09:00:40.718 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:40.720 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c9c0377c0 msgr2=0x7f3c9c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:40.720 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 --2- 172.21.15.103:0/4047556250 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c9c0377c0 0x7f3c9c039cd0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb812f250 msgr2=0x7f3cb812b5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 --2- 172.21.15.103:0/4047556250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb812f250 0x7f3cb812b5a0 crc :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 shutdown_connections 2023-07-02T09:00:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 --2- 172.21.15.103:0/4047556250 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c9c0377c0 0x7f3c9c039cd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 --2- 172.21.15.103:0/4047556250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cb812f250 0x7f3cb812b5a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 >> 172.21.15.103:0/4047556250 conn(0x7f3cb809d810 msgr2=0x7f3cb812e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:40.722 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 shutdown_connections 2023-07-02T09:00:40.722 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:40.719+0000 7f3cbede0700 1 -- 172.21.15.103:0/4047556250 wait complete. 2023-07-02T09:00:40.733 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:40.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2023-07-02T09:00:41.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.865+0000 7f7228e92700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.870+0000 7f7228e92700 1 Processor -- start 2023-07-02T09:00:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.870+0000 7f7228e92700 1 -- start start 2023-07-02T09:00:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f7228e92700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7224099520 0x7f7224135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f7228e92700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722406f8b0 con 0x7f7224098a00 2023-07-02T09:00:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f7228e92700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7224072510 con 0x7f7224099520 2023-07-02T09:00:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f7221d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7224099520 0x7f7224135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f722259c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7224098a00 0x7f7224098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41610/0 (socket says 172.21.15.103:41610) 2023-07-02T09:00:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f722259c700 1 -- 172.21.15.103:0/195340379 learned_addr learned my addr 172.21.15.103:0/195340379 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f7221d9b700 1 -- 172.21.15.103:0/195340379 >> v1:172.21.15.103:6789/0 conn(0x7f7224098a00 legacy=0x7f7224098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f7221d9b700 1 -- 172.21.15.103:0/195340379 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722409aec0 con 0x7f7224099520 2023-07-02T09:00:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.871+0000 7f722159a700 1 -- 172.21.15.103:0/195340379 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268908023 0 0) 0x7f722406f8b0 con 0x7f7224098a00 2023-07-02T09:00:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f7221d9b700 1 --2- 172.21.15.103:0/195340379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224099520 0x7f7224135ca0 crc :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637153debcaec810 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f722159a700 1 -- 172.21.15.103:0/195340379 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7214009cd0 con 0x7f7224099520 2023-07-02T09:00:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f722159a700 1 -- 172.21.15.103:0/195340379 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7214009e30 con 0x7f7224099520 2023-07-02T09:00:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f722159a700 1 -- 172.21.15.103:0/195340379 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f721400e360 con 0x7f7224099520 2023-07-02T09:00:41.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f7228e92700 1 -- 172.21.15.103:0/195340379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224099520 msgr2=0x7f7224135ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:41.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f7228e92700 1 --2- 172.21.15.103:0/195340379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224099520 0x7f7224135ca0 crc :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:41.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f7228e92700 1 -- 172.21.15.103:0/195340379 shutdown_connections 2023-07-02T09:00:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f7228e92700 1 --2- 172.21.15.103:0/195340379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224099520 0x7f7224135ca0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f7228e92700 1 -- 172.21.15.103:0/195340379 >> 172.21.15.103:0/195340379 conn(0x7f722409d810 msgr2=0x7f722409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f7228e92700 1 -- 172.21.15.103:0/195340379 shutdown_connections 2023-07-02T09:00:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.872+0000 7f7228e92700 1 -- 172.21.15.103:0/195340379 wait complete. 2023-07-02T09:00:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.873+0000 7f7228e92700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.874+0000 7f7228e92700 1 Processor -- start 2023-07-02T09:00:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.874+0000 7f7228e92700 1 -- start start 2023-07-02T09:00:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.874+0000 7f7228e92700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224098a00 0x7f72241c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.874+0000 7f7228e92700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7224122bc0 con 0x7f7224098a00 2023-07-02T09:00:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.874+0000 7f722259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224098a00 0x7f72241c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.874+0000 7f722259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224098a00 0x7f72241c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48288/0 (socket says 172.21.15.103:48288) 2023-07-02T09:00:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.874+0000 7f722259c700 1 -- 172.21.15.103:0/454293735 learned_addr learned my addr 172.21.15.103:0/454293735 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.875+0000 7f722259c700 1 -- 172.21.15.103:0/454293735 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722406d4f0 con 0x7f7224098a00 2023-07-02T09:00:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.875+0000 7f722259c700 1 --2- 172.21.15.103:0/454293735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224098a00 0x7f72241c6b70 crc :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.875+0000 7f7212ffd700 1 -- 172.21.15.103:0/454293735 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7218004b80 con 0x7f7224098a00 2023-07-02T09:00:41.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.875+0000 7f7212ffd700 1 -- 172.21.15.103:0/454293735 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7218004ce0 con 0x7f7224098a00 2023-07-02T09:00:41.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.875+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722406d150 con 0x7f7224098a00 2023-07-02T09:00:41.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.875+0000 7f7212ffd700 1 -- 172.21.15.103:0/454293735 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f721800c550 con 0x7f7224098a00 2023-07-02T09:00:41.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.875+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7224121f10 con 0x7f7224098a00 2023-07-02T09:00:41.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.876+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7224061ea0 con 0x7f7224098a00 2023-07-02T09:00:41.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.876+0000 7f7212ffd700 1 -- 172.21.15.103:0/454293735 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f721800c700 con 0x7f7224098a00 2023-07-02T09:00:41.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.876+0000 7f7212ffd700 1 --2- 172.21.15.103:0/454293735 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7204037770 0x7f7204039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:41.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.876+0000 7f7212ffd700 1 -- 172.21.15.103:0/454293735 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7218046ab0 con 0x7f7224098a00 2023-07-02T09:00:41.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.876+0000 7f7221d9b700 1 --2- 172.21.15.103:0/454293735 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7204037770 0x7f7204039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:41.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.877+0000 7f7221d9b700 1 --2- 172.21.15.103:0/454293735 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7204037770 0x7f7204039c80 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:41.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:41.881+0000 7f7212ffd700 1 -- 172.21.15.103:0/454293735 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7218009a80 con 0x7f7224098a00 2023-07-02T09:00:42.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.075+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f722406d2e0 con 0x7f7224098a00 2023-07-02T09:00:42.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.076+0000 7f7212ffd700 1 -- 172.21.15.103:0/454293735 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f72180098a0 con 0x7f7224098a00 2023-07-02T09:00:42.078 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:42.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7204037770 msgr2=0x7f7204039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 --2- 172.21.15.103:0/454293735 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7204037770 0x7f7204039c80 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224098a00 msgr2=0x7f72241c6b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 --2- 172.21.15.103:0/454293735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224098a00 0x7f72241c6b70 crc :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 shutdown_connections 2023-07-02T09:00:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 --2- 172.21.15.103:0/454293735 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7204037770 0x7f7204039c80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 --2- 172.21.15.103:0/454293735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7224098a00 0x7f72241c6b70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 >> 172.21.15.103:0/454293735 conn(0x7f722409d810 msgr2=0x7f72241306f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:42.081 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 shutdown_connections 2023-07-02T09:00:42.081 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:42.078+0000 7f7228e92700 1 -- 172.21.15.103:0/454293735 wait complete. 2023-07-02T09:00:42.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:43.094 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:43.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.226+0000 7f908f313700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:43.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.231+0000 7f908f313700 1 Processor -- start 2023-07-02T09:00:43.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.232+0000 7f908f313700 1 -- start start 2023-07-02T09:00:43.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.232+0000 7f908f313700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f908809d8a0 0x7f908809dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:43.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.232+0000 7f908f313700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908809f2e0 con 0x7f908809e1f0 2023-07-02T09:00:43.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.232+0000 7f908f313700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9088099090 con 0x7f908809d8a0 2023-07-02T09:00:43.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f908d0af700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f908809d8a0 0x7f908809dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:43.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f908c8ae700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f908809e1f0 0x7f9088132a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41618/0 (socket says 172.21.15.103:41618) 2023-07-02T09:00:43.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f908d0af700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f908809d8a0 0x7f908809dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48300/0 (socket says 172.21.15.103:48300) 2023-07-02T09:00:43.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f908c8ae700 1 -- 172.21.15.103:0/881924575 learned_addr learned my addr 172.21.15.103:0/881924575 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:43.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f908d0af700 1 -- 172.21.15.103:0/881924575 >> v1:172.21.15.103:6789/0 conn(0x7f908809e1f0 legacy=0x7f9088132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:43.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f908d0af700 1 -- 172.21.15.103:0/881924575 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908809a000 con 0x7f908809d8a0 2023-07-02T09:00:43.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f907ffff700 1 -- 172.21.15.103:0/881924575 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943775289 0 0) 0x7f908809f2e0 con 0x7f908809e1f0 2023-07-02T09:00:43.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f908d0af700 1 --2- 172.21.15.103:0/881924575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f908809dcb0 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed40f9ae54903d71 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:43.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f907ffff700 1 -- 172.21.15.103:0/881924575 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9078009cd0 con 0x7f908809d8a0 2023-07-02T09:00:43.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f907ffff700 1 -- 172.21.15.103:0/881924575 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9078009e30 con 0x7f908809d8a0 2023-07-02T09:00:43.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.233+0000 7f907ffff700 1 -- 172.21.15.103:0/881924575 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f907800e360 con 0x7f908809d8a0 2023-07-02T09:00:43.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.234+0000 7f908f313700 1 -- 172.21.15.103:0/881924575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 msgr2=0x7f908809dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:43.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.234+0000 7f908f313700 1 --2- 172.21.15.103:0/881924575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f908809dcb0 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:43.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.234+0000 7f908f313700 1 -- 172.21.15.103:0/881924575 shutdown_connections 2023-07-02T09:00:43.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.234+0000 7f908f313700 1 --2- 172.21.15.103:0/881924575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f908809dcb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:43.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.234+0000 7f908f313700 1 -- 172.21.15.103:0/881924575 >> 172.21.15.103:0/881924575 conn(0x7f90880a3280 msgr2=0x7f90880a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:43.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.234+0000 7f908f313700 1 -- 172.21.15.103:0/881924575 shutdown_connections 2023-07-02T09:00:43.238 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.234+0000 7f908f313700 1 -- 172.21.15.103:0/881924575 wait complete. 2023-07-02T09:00:43.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.234+0000 7f908f313700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:43.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.235+0000 7f908f313700 1 Processor -- start 2023-07-02T09:00:43.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.235+0000 7f908f313700 1 -- start start 2023-07-02T09:00:43.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f908f313700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f90881c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:43.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f908f313700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90881c34e0 con 0x7f908809d8a0 2023-07-02T09:00:43.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f908d0af700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f90881c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:43.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f908d0af700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f90881c3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48306/0 (socket says 172.21.15.103:48306) 2023-07-02T09:00:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f908d0af700 1 -- 172.21.15.103:0/666821409 learned_addr learned my addr 172.21.15.103:0/666821409 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f908d0af700 1 -- 172.21.15.103:0/666821409 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90881a91d0 con 0x7f908809d8a0 2023-07-02T09:00:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f908d0af700 1 --2- 172.21.15.103:0/666821409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f90881c3a20 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f907dffb700 1 -- 172.21.15.103:0/666821409 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9078013530 con 0x7f908809d8a0 2023-07-02T09:00:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f907dffb700 1 -- 172.21.15.103:0/666821409 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9078013690 con 0x7f908809d8a0 2023-07-02T09:00:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90881c3710 con 0x7f908809d8a0 2023-07-02T09:00:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.236+0000 7f907dffb700 1 -- 172.21.15.103:0/666821409 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9078013930 con 0x7f908809d8a0 2023-07-02T09:00:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.237+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90881a9830 con 0x7f908809d8a0 2023-07-02T09:00:43.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.237+0000 7f907dffb700 1 -- 172.21.15.103:0/666821409 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f9078013ae0 con 0x7f908809d8a0 2023-07-02T09:00:43.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.237+0000 7f907dffb700 1 --2- 172.21.15.103:0/666821409 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f906c0377b0 0x7f906c039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.237+0000 7f907dffb700 1 -- 172.21.15.103:0/666821409 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9078018040 con 0x7f908809d8a0 2023-07-02T09:00:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.238+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9088061ea0 con 0x7f908809d8a0 2023-07-02T09:00:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.238+0000 7f908c8ae700 1 --2- 172.21.15.103:0/666821409 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f906c0377b0 0x7f906c039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.238+0000 7f908c8ae700 1 --2- 172.21.15.103:0/666821409 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f906c0377b0 0x7f906c039cc0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:43.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.243+0000 7f907dffb700 1 -- 172.21.15.103:0/666821409 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f9078018d80 con 0x7f908809d8a0 2023-07-02T09:00:43.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.435+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9088073ca0 con 0x7f908809d8a0 2023-07-02T09:00:43.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.435+0000 7f907dffb700 1 -- 172.21.15.103:0/666821409 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f9078018b50 con 0x7f908809d8a0 2023-07-02T09:00:43.437 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:43.439 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f906c0377b0 msgr2=0x7f906c039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:43.439 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 --2- 172.21.15.103:0/666821409 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f906c0377b0 0x7f906c039cc0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:43.439 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 msgr2=0x7f90881c3a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:43.439 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 --2- 172.21.15.103:0/666821409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f90881c3a20 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:43.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 shutdown_connections 2023-07-02T09:00:43.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 --2- 172.21.15.103:0/666821409 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f906c0377b0 0x7f906c039cc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:43.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 --2- 172.21.15.103:0/666821409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f908809d8a0 0x7f90881c3a20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:43.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 >> 172.21.15.103:0/666821409 conn(0x7f90880a3280 msgr2=0x7f908806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:43.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 shutdown_connections 2023-07-02T09:00:43.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:43.438+0000 7f908f313700 1 -- 172.21.15.103:0/666821409 wait complete. 2023-07-02T09:00:43.452 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:44.453 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.591+0000 7f3c6bcde700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.596+0000 7f3c6bcde700 1 Processor -- start 2023-07-02T09:00:44.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.596+0000 7f3c6bcde700 1 -- start start 2023-07-02T09:00:44.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.597+0000 7f3c6bcde700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c64132f50 0x7f3c64133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:44.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.597+0000 7f3c6bcde700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6409e260 con 0x7f3c6409c580 2023-07-02T09:00:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.597+0000 7f3c6bcde700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c640a1640 con 0x7f3c64132f50 2023-07-02T09:00:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.597+0000 7f3c69a7a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c64132f50 0x7f3c64133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.597+0000 7f3c69279700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3c6409c580 0x7f3c6409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42066/0 (socket says 172.21.15.103:42066) 2023-07-02T09:00:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.597+0000 7f3c69279700 1 -- 172.21.15.103:0/4114803731 learned_addr learned my addr 172.21.15.103:0/4114803731 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:44.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.597+0000 7f3c69a7a700 1 -- 172.21.15.103:0/4114803731 >> v1:172.21.15.103:6789/0 conn(0x7f3c6409c580 legacy=0x7f3c6409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:44.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c68a78700 1 -- 172.21.15.103:0/4114803731 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747166191 0 0) 0x7f3c6409e260 con 0x7f3c6409c580 2023-07-02T09:00:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c69a7a700 1 -- 172.21.15.103:0/4114803731 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c64132a20 con 0x7f3c64132f50 2023-07-02T09:00:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c69a7a700 1 --2- 172.21.15.103:0/4114803731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64132f50 0x7f3c64133360 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fce34961690d80 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c68a78700 1 -- 172.21.15.103:0/4114803731 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3c54009cd0 con 0x7f3c64132f50 2023-07-02T09:00:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c68a78700 1 -- 172.21.15.103:0/4114803731 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3c54009e30 con 0x7f3c64132f50 2023-07-02T09:00:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c68a78700 1 -- 172.21.15.103:0/4114803731 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3c5400e360 con 0x7f3c64132f50 2023-07-02T09:00:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c6bcde700 1 -- 172.21.15.103:0/4114803731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64132f50 msgr2=0x7f3c64133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c6bcde700 1 --2- 172.21.15.103:0/4114803731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64132f50 0x7f3c64133360 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c6bcde700 1 -- 172.21.15.103:0/4114803731 shutdown_connections 2023-07-02T09:00:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.598+0000 7f3c6bcde700 1 --2- 172.21.15.103:0/4114803731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c64132f50 0x7f3c64133360 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.599+0000 7f3c6bcde700 1 -- 172.21.15.103:0/4114803731 >> 172.21.15.103:0/4114803731 conn(0x7f3c640a2080 msgr2=0x7f3c640a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.599+0000 7f3c6bcde700 1 -- 172.21.15.103:0/4114803731 shutdown_connections 2023-07-02T09:00:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.599+0000 7f3c6bcde700 1 -- 172.21.15.103:0/4114803731 wait complete. 2023-07-02T09:00:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.599+0000 7f3c6bcde700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.600+0000 7f3c6bcde700 1 Processor -- start 2023-07-02T09:00:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.600+0000 7f3c6bcde700 1 -- start start 2023-07-02T09:00:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.600+0000 7f3c6bcde700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c6409c580 0x7f3c641baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.600+0000 7f3c6bcde700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c641ba610 con 0x7f3c6409c580 2023-07-02T09:00:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c69a7a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c6409c580 0x7f3c641baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c69a7a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c6409c580 0x7f3c641baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34738/0 (socket says 172.21.15.103:34738) 2023-07-02T09:00:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c69a7a700 1 -- 172.21.15.103:0/1331270832 learned_addr learned my addr 172.21.15.103:0/1331270832 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c69a7a700 1 -- 172.21.15.103:0/1331270832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6406dfc0 con 0x7f3c6409c580 2023-07-02T09:00:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c69a7a700 1 --2- 172.21.15.103:0/1331270832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c6409c580 0x7f3c641baa60 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c5a7fc700 1 -- 172.21.15.103:0/1331270832 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3c54013530 con 0x7f3c6409c580 2023-07-02T09:00:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c5a7fc700 1 -- 172.21.15.103:0/1331270832 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3c54013690 con 0x7f3c6409c580 2023-07-02T09:00:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c641ba270 con 0x7f3c6409c580 2023-07-02T09:00:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c5a7fc700 1 -- 172.21.15.103:0/1331270832 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3c54013930 con 0x7f3c6409c580 2023-07-02T09:00:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.601+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6406dba0 con 0x7f3c6409c580 2023-07-02T09:00:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.602+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c641ba460 con 0x7f3c6409c580 2023-07-02T09:00:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.602+0000 7f3c5a7fc700 1 -- 172.21.15.103:0/1331270832 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3c54013ae0 con 0x7f3c6409c580 2023-07-02T09:00:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.602+0000 7f3c5a7fc700 1 --2- 172.21.15.103:0/1331270832 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c480377b0 0x7f3c48039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.602+0000 7f3c5a7fc700 1 -- 172.21.15.103:0/1331270832 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3c54018040 con 0x7f3c6409c580 2023-07-02T09:00:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.603+0000 7f3c69279700 1 --2- 172.21.15.103:0/1331270832 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c480377b0 0x7f3c48039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:44.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.603+0000 7f3c69279700 1 --2- 172.21.15.103:0/1331270832 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c480377b0 0x7f3c48039cc0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.607+0000 7f3c5a7fc700 1 -- 172.21.15.103:0/1331270832 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3c54018e00 con 0x7f3c6409c580 2023-07-02T09:00:44.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.797+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3c64061ea0 con 0x7f3c6409c580 2023-07-02T09:00:44.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.798+0000 7f3c5a7fc700 1 -- 172.21.15.103:0/1331270832 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3c54010e60 con 0x7f3c6409c580 2023-07-02T09:00:44.799 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:44.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c480377b0 msgr2=0x7f3c48039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:44.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 --2- 172.21.15.103:0/1331270832 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c480377b0 0x7f3c48039cc0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:44.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c6409c580 msgr2=0x7f3c641baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:44.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 --2- 172.21.15.103:0/1331270832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c6409c580 0x7f3c641baa60 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:44.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 shutdown_connections 2023-07-02T09:00:44.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 --2- 172.21.15.103:0/1331270832 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3c480377b0 0x7f3c48039cc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:44.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 --2- 172.21.15.103:0/1331270832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c6409c580 0x7f3c641baa60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:44.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 >> 172.21.15.103:0/1331270832 conn(0x7f3c640a2080 msgr2=0x7f3c64130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:44.803 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 shutdown_connections 2023-07-02T09:00:44.803 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:44.800+0000 7f3c6bcde700 1 -- 172.21.15.103:0/1331270832 wait complete. 2023-07-02T09:00:44.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:45.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:45.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.955+0000 7feb3dd6c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:45.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.960+0000 7feb3dd6c700 1 Processor -- start 2023-07-02T09:00:45.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.960+0000 7feb3dd6c700 1 -- start start 2023-07-02T09:00:45.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb3dd6c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb38132030 0x7feb38134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:45.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb3dd6c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3806df60 con 0x7feb3812f250 2023-07-02T09:00:45.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb3dd6c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb3809a710 con 0x7feb38132030 2023-07-02T09:00:45.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb36ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb38132030 0x7feb38134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:45.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb36ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb38132030 0x7feb38134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34746/0 (socket says 172.21.15.103:34746) 2023-07-02T09:00:45.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb377fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7feb3812f250 0x7feb3812f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42082/0 (socket says 172.21.15.103:42082) 2023-07-02T09:00:45.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb36ffd700 1 -- 172.21.15.103:0/4251564568 learned_addr learned my addr 172.21.15.103:0/4251564568 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:45.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb36ffd700 1 -- 172.21.15.103:0/4251564568 >> v1:172.21.15.103:6789/0 conn(0x7feb3812f250 legacy=0x7feb3812f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:45.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb36ffd700 1 -- 172.21.15.103:0/4251564568 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3812ed20 con 0x7feb38132030 2023-07-02T09:00:45.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb36ffd700 1 --2- 172.21.15.103:0/4251564568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb38132030 0x7feb38134450 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b13c06e4f4fced server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:45.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb367fc700 1 -- 172.21.15.103:0/4251564568 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7feb2800ad20 con 0x7feb38132030 2023-07-02T09:00:45.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb367fc700 1 -- 172.21.15.103:0/4251564568 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7feb2800ae80 con 0x7feb38132030 2023-07-02T09:00:45.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.961+0000 7feb367fc700 1 -- 172.21.15.103:0/4251564568 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7feb2800f470 con 0x7feb38132030 2023-07-02T09:00:45.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.962+0000 7feb3dd6c700 1 -- 172.21.15.103:0/4251564568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb38132030 msgr2=0x7feb38134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:45.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.962+0000 7feb3dd6c700 1 --2- 172.21.15.103:0/4251564568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb38132030 0x7feb38134450 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:45.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.962+0000 7feb3dd6c700 1 -- 172.21.15.103:0/4251564568 shutdown_connections 2023-07-02T09:00:45.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.962+0000 7feb3dd6c700 1 --2- 172.21.15.103:0/4251564568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb38132030 0x7feb38134450 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:45.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.962+0000 7feb3dd6c700 1 -- 172.21.15.103:0/4251564568 >> 172.21.15.103:0/4251564568 conn(0x7feb3809d810 msgr2=0x7feb3809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:45.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.962+0000 7feb3dd6c700 1 -- 172.21.15.103:0/4251564568 shutdown_connections 2023-07-02T09:00:45.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.962+0000 7feb3dd6c700 1 -- 172.21.15.103:0/4251564568 wait complete. 2023-07-02T09:00:45.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.962+0000 7feb3dd6c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:45.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.963+0000 7feb3dd6c700 1 Processor -- start 2023-07-02T09:00:45.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.963+0000 7feb3dd6c700 1 -- start start 2023-07-02T09:00:45.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.964+0000 7feb3dd6c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb3812f250 0x7feb38124a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:45.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.964+0000 7feb3dd6c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb38099af0 con 0x7feb3812f250 2023-07-02T09:00:45.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.964+0000 7feb377fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb3812f250 0x7feb38124a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:45.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.964+0000 7feb377fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb3812f250 0x7feb38124a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34750/0 (socket says 172.21.15.103:34750) 2023-07-02T09:00:45.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.964+0000 7feb377fe700 1 -- 172.21.15.103:0/363705116 learned_addr learned my addr 172.21.15.103:0/363705116 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:45.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.964+0000 7feb377fe700 1 -- 172.21.15.103:0/363705116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38071d80 con 0x7feb3812f250 2023-07-02T09:00:45.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.964+0000 7feb377fe700 1 --2- 172.21.15.103:0/363705116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb3812f250 0x7feb38124a10 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:45.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.965+0000 7feb1ffff700 1 -- 172.21.15.103:0/363705116 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7feb2c0059a0 con 0x7feb3812f250 2023-07-02T09:00:45.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.965+0000 7feb1ffff700 1 -- 172.21.15.103:0/363705116 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7feb2c005b00 con 0x7feb3812f250 2023-07-02T09:00:45.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.965+0000 7feb1ffff700 1 -- 172.21.15.103:0/363705116 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7feb2c005da0 con 0x7feb3812f250 2023-07-02T09:00:45.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.965+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb380704e0 con 0x7feb3812f250 2023-07-02T09:00:45.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.965+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb38070080 con 0x7feb3812f250 2023-07-02T09:00:45.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.965+0000 7feb1ffff700 1 -- 172.21.15.103:0/363705116 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7feb2c0070b0 con 0x7feb3812f250 2023-07-02T09:00:45.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.966+0000 7feb1ffff700 1 --2- 172.21.15.103:0/363705116 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb240377c0 0x7feb24039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:45.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.966+0000 7feb1ffff700 1 -- 172.21.15.103:0/363705116 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7feb2c046b00 con 0x7feb3812f250 2023-07-02T09:00:45.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.966+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3812b4b0 con 0x7feb3812f250 2023-07-02T09:00:45.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.966+0000 7feb36ffd700 1 --2- 172.21.15.103:0/363705116 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb240377c0 0x7feb24039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:45.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.967+0000 7feb36ffd700 1 --2- 172.21.15.103:0/363705116 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb240377c0 0x7feb24039cd0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:45.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:45.971+0000 7feb1ffff700 1 -- 172.21.15.103:0/363705116 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7feb2c00a670 con 0x7feb3812f250 2023-07-02T09:00:46.164 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.163+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7feb3809dc20 con 0x7feb3812f250 2023-07-02T09:00:46.164 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.163+0000 7feb1ffff700 1 -- 172.21.15.103:0/363705116 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7feb2c00fe20 con 0x7feb3812f250 2023-07-02T09:00:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:46.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.165+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb240377c0 msgr2=0x7feb24039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:46.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.165+0000 7feb3dd6c700 1 --2- 172.21.15.103:0/363705116 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb240377c0 0x7feb24039cd0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:46.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.165+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb3812f250 msgr2=0x7feb38124a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:46.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.165+0000 7feb3dd6c700 1 --2- 172.21.15.103:0/363705116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb3812f250 0x7feb38124a10 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:46.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.165+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 shutdown_connections 2023-07-02T09:00:46.167 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.165+0000 7feb3dd6c700 1 --2- 172.21.15.103:0/363705116 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7feb240377c0 0x7feb24039cd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:46.168 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.165+0000 7feb3dd6c700 1 --2- 172.21.15.103:0/363705116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb3812f250 0x7feb38124a10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:46.168 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.165+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 >> 172.21.15.103:0/363705116 conn(0x7feb3809d810 msgr2=0x7feb38128e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:46.168 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.166+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 shutdown_connections 2023-07-02T09:00:46.168 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:46.166+0000 7feb3dd6c700 1 -- 172.21.15.103:0/363705116 wait complete. 2023-07-02T09:00:46.180 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:46.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~61s 2023-07-02T09:00:47.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:47.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.316+0000 7f783c2e5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:47.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.321+0000 7f783c2e5700 1 Processor -- start 2023-07-02T09:00:47.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.321+0000 7f783c2e5700 1 -- start start 2023-07-02T09:00:47.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f783c2e5700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7834132f50 0x7f7834133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:47.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f783c2e5700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783409e260 con 0x7f783409c580 2023-07-02T09:00:47.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f783c2e5700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78340a1640 con 0x7f7834132f50 2023-07-02T09:00:47.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f783a081700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7834132f50 0x7f7834133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:47.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f7839880700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f783409c580 0x7f783409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42090/0 (socket says 172.21.15.103:42090) 2023-07-02T09:00:47.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f7839880700 1 -- 172.21.15.103:0/4169571189 learned_addr learned my addr 172.21.15.103:0/4169571189 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:47.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f783a081700 1 -- 172.21.15.103:0/4169571189 >> v1:172.21.15.103:6789/0 conn(0x7f783409c580 legacy=0x7f783409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:47.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f783a081700 1 -- 172.21.15.103:0/4169571189 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7834132a20 con 0x7f7834132f50 2023-07-02T09:00:47.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.322+0000 7f783a081700 1 --2- 172.21.15.103:0/4169571189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7834132f50 0x7f7834133360 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290edefe7a69a03 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:47.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783907f700 1 -- 172.21.15.103:0/4169571189 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7824009cd0 con 0x7f7834132f50 2023-07-02T09:00:47.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783907f700 1 -- 172.21.15.103:0/4169571189 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7824009e30 con 0x7f7834132f50 2023-07-02T09:00:47.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783907f700 1 -- 172.21.15.103:0/4169571189 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f782400e360 con 0x7f7834132f50 2023-07-02T09:00:47.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783c2e5700 1 -- 172.21.15.103:0/4169571189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7834132f50 msgr2=0x7f7834133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:47.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783c2e5700 1 --2- 172.21.15.103:0/4169571189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7834132f50 0x7f7834133360 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:47.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783c2e5700 1 -- 172.21.15.103:0/4169571189 shutdown_connections 2023-07-02T09:00:47.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783c2e5700 1 --2- 172.21.15.103:0/4169571189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7834132f50 0x7f7834133360 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:47.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783c2e5700 1 -- 172.21.15.103:0/4169571189 >> 172.21.15.103:0/4169571189 conn(0x7f78340a2080 msgr2=0x7f78340a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:47.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783c2e5700 1 -- 172.21.15.103:0/4169571189 shutdown_connections 2023-07-02T09:00:47.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.323+0000 7f783c2e5700 1 -- 172.21.15.103:0/4169571189 wait complete. 2023-07-02T09:00:47.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.324+0000 7f783c2e5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:47.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.325+0000 7f783c2e5700 1 Processor -- start 2023-07-02T09:00:47.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.325+0000 7f783c2e5700 1 -- start start 2023-07-02T09:00:47.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.325+0000 7f783c2e5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f783409c580 0x7f783412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:47.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.325+0000 7f783c2e5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783406f9d0 con 0x7f783409c580 2023-07-02T09:00:47.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.325+0000 7f783a081700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f783409c580 0x7f783412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:47.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.325+0000 7f783a081700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f783409c580 0x7f783412b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34756/0 (socket says 172.21.15.103:34756) 2023-07-02T09:00:47.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.325+0000 7f783a081700 1 -- 172.21.15.103:0/3643128891 learned_addr learned my addr 172.21.15.103:0/3643128891 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:47.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.326+0000 7f783a081700 1 -- 172.21.15.103:0/3643128891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7834133a00 con 0x7f783409c580 2023-07-02T09:00:47.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.326+0000 7f783a081700 1 --2- 172.21.15.103:0/3643128891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f783409c580 0x7f783412b7b0 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:47.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.326+0000 7f782affd700 1 -- 172.21.15.103:0/3643128891 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7824013530 con 0x7f783409c580 2023-07-02T09:00:47.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.326+0000 7f782affd700 1 -- 172.21.15.103:0/3643128891 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7824013690 con 0x7f783409c580 2023-07-02T09:00:47.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.326+0000 7f782affd700 1 -- 172.21.15.103:0/3643128891 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7824013930 con 0x7f783409c580 2023-07-02T09:00:47.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.326+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783406d160 con 0x7f783409c580 2023-07-02T09:00:47.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.326+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783406da70 con 0x7f783409c580 2023-07-02T09:00:47.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.327+0000 7f782affd700 1 -- 172.21.15.103:0/3643128891 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7824013ae0 con 0x7f783409c580 2023-07-02T09:00:47.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.327+0000 7f782affd700 1 --2- 172.21.15.103:0/3643128891 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78180377c0 0x7f7818039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:47.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.327+0000 7f782affd700 1 -- 172.21.15.103:0/3643128891 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7824018040 con 0x7f783409c580 2023-07-02T09:00:47.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.327+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7834061ea0 con 0x7f783409c580 2023-07-02T09:00:47.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.327+0000 7f7839880700 1 --2- 172.21.15.103:0/3643128891 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78180377c0 0x7f7818039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:47.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.328+0000 7f7839880700 1 --2- 172.21.15.103:0/3643128891 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78180377c0 0x7f7818039cd0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:47.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.332+0000 7f782affd700 1 -- 172.21.15.103:0/3643128891 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7824018dd0 con 0x7f783409c580 2023-07-02T09:00:47.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.524+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f78340725b0 con 0x7f783409c580 2023-07-02T09:00:47.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.524+0000 7f782affd700 1 -- 172.21.15.103:0/3643128891 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f78240189f0 con 0x7f783409c580 2023-07-02T09:00:47.526 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:47.528 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78180377c0 msgr2=0x7f7818039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:47.528 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 --2- 172.21.15.103:0/3643128891 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78180377c0 0x7f7818039cd0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:47.528 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f783409c580 msgr2=0x7f783412b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:47.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 --2- 172.21.15.103:0/3643128891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f783409c580 0x7f783412b7b0 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:47.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 shutdown_connections 2023-07-02T09:00:47.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 --2- 172.21.15.103:0/3643128891 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78180377c0 0x7f7818039cd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:47.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 --2- 172.21.15.103:0/3643128891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f783409c580 0x7f783412b7b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:47.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 >> 172.21.15.103:0/3643128891 conn(0x7f78340a2080 msgr2=0x7f78341babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:47.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 shutdown_connections 2023-07-02T09:00:47.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:47.527+0000 7f783c2e5700 1 -- 172.21.15.103:0/3643128891 wait complete. 2023-07-02T09:00:47.541 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:48.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:48.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.680+0000 7f429a8df700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:48.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.685+0000 7f429a8df700 1 Processor -- start 2023-07-02T09:00:48.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.686+0000 7f429a8df700 1 -- start start 2023-07-02T09:00:48.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.686+0000 7f429a8df700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4294133360 0x7f429409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:48.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.686+0000 7f429a8df700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429409e260 con 0x7f4294132f50 2023-07-02T09:00:48.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.686+0000 7f429a8df700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940a1640 con 0x7f4294133360 2023-07-02T09:00:48.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.686+0000 7f4293fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4294132f50 0x7f429409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42098/0 (socket says 172.21.15.103:42098) 2023-07-02T09:00:48.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.686+0000 7f4293fff700 1 -- 172.21.15.103:0/1772627538 learned_addr learned my addr 172.21.15.103:0/1772627538 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:48.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.686+0000 7f42937fe700 1 --2- 172.21.15.103:0/1772627538 >> v2:172.21.15.103:3300/0 conn(0x7f4294133360 0x7f429409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:48.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.687+0000 7f42937fe700 1 -- 172.21.15.103:0/1772627538 >> v1:172.21.15.103:6789/0 conn(0x7f4294132f50 legacy=0x7f429409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:48.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.687+0000 7f42937fe700 1 -- 172.21.15.103:0/1772627538 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4294132a20 con 0x7f4294133360 2023-07-02T09:00:48.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.687+0000 7f4292ffd700 1 -- 172.21.15.103:0/1772627538 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164953438 0 0) 0x7f429409e260 con 0x7f4294132f50 2023-07-02T09:00:48.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.687+0000 7f42937fe700 1 --2- 172.21.15.103:0/1772627538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294133360 0x7f429409cc90 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4654c185b3c0359 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:48.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.687+0000 7f4292ffd700 1 -- 172.21.15.103:0/1772627538 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4288009cd0 con 0x7f4294133360 2023-07-02T09:00:48.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.687+0000 7f4292ffd700 1 -- 172.21.15.103:0/1772627538 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4288009e30 con 0x7f4294133360 2023-07-02T09:00:48.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.687+0000 7f4292ffd700 1 -- 172.21.15.103:0/1772627538 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f428800e360 con 0x7f4294133360 2023-07-02T09:00:48.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.688+0000 7f429a8df700 1 -- 172.21.15.103:0/1772627538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294133360 msgr2=0x7f429409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:48.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.688+0000 7f429a8df700 1 --2- 172.21.15.103:0/1772627538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294133360 0x7f429409cc90 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:48.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.688+0000 7f429a8df700 1 -- 172.21.15.103:0/1772627538 shutdown_connections 2023-07-02T09:00:48.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.688+0000 7f429a8df700 1 --2- 172.21.15.103:0/1772627538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294133360 0x7f429409cc90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:48.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.688+0000 7f429a8df700 1 -- 172.21.15.103:0/1772627538 >> 172.21.15.103:0/1772627538 conn(0x7f42940a2080 msgr2=0x7f42940a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:48.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.688+0000 7f429a8df700 1 -- 172.21.15.103:0/1772627538 shutdown_connections 2023-07-02T09:00:48.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.688+0000 7f429a8df700 1 -- 172.21.15.103:0/1772627538 wait complete. 2023-07-02T09:00:48.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.688+0000 7f429a8df700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:48.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.689+0000 7f429a8df700 1 Processor -- start 2023-07-02T09:00:48.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.689+0000 7f429a8df700 1 -- start start 2023-07-02T09:00:48.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f429a8df700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294132f50 0x7f42941ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:48.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f429a8df700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429406d370 con 0x7f4294132f50 2023-07-02T09:00:48.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f4293fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294132f50 0x7f42941ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:48.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f4293fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294132f50 0x7f42941ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34780/0 (socket says 172.21.15.103:34780) 2023-07-02T09:00:48.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f4293fff700 1 -- 172.21.15.103:0/1440145414 learned_addr learned my addr 172.21.15.103:0/1440145414 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:48.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f4293fff700 1 -- 172.21.15.103:0/1440145414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429406dde0 con 0x7f4294132f50 2023-07-02T09:00:48.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f4293fff700 1 --2- 172.21.15.103:0/1440145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294132f50 0x7f42941ba9b0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:48.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f4290ff9700 1 -- 172.21.15.103:0/1440145414 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4284004b80 con 0x7f4294132f50 2023-07-02T09:00:48.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f4290ff9700 1 -- 172.21.15.103:0/1440145414 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4284004ce0 con 0x7f4294132f50 2023-07-02T09:00:48.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42941ba4f0 con 0x7f4294132f50 2023-07-02T09:00:48.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f4290ff9700 1 -- 172.21.15.103:0/1440145414 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f428400c4f0 con 0x7f4294132f50 2023-07-02T09:00:48.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.690+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4294071e40 con 0x7f4294132f50 2023-07-02T09:00:48.695 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.692+0000 7f4290ff9700 1 -- 172.21.15.103:0/1440145414 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f428400c650 con 0x7f4294132f50 2023-07-02T09:00:48.695 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.692+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4294061ea0 con 0x7f4294132f50 2023-07-02T09:00:48.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.692+0000 7f4290ff9700 1 --2- 172.21.15.103:0/1440145414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f42800377c0 0x7f4280039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:48.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.692+0000 7f4290ff9700 1 -- 172.21.15.103:0/1440145414 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4284046970 con 0x7f4294132f50 2023-07-02T09:00:48.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.692+0000 7f42937fe700 1 --2- 172.21.15.103:0/1440145414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f42800377c0 0x7f4280039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:48.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.693+0000 7f42937fe700 1 --2- 172.21.15.103:0/1440145414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f42800377c0 0x7f4280039cd0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:48.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.696+0000 7f4290ff9700 1 -- 172.21.15.103:0/1440145414 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f4284009c80 con 0x7f4294132f50 2023-07-02T09:00:48.891 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.890+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f42940725b0 con 0x7f4294132f50 2023-07-02T09:00:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.891+0000 7f4290ff9700 1 -- 172.21.15.103:0/1440145414 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f42840098a0 con 0x7f4294132f50 2023-07-02T09:00:48.892 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:48.894 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f42800377c0 msgr2=0x7f4280039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:48.894 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 --2- 172.21.15.103:0/1440145414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f42800377c0 0x7f4280039cd0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:48.894 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294132f50 msgr2=0x7f42941ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:48.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 --2- 172.21.15.103:0/1440145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294132f50 0x7f42941ba9b0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:48.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 shutdown_connections 2023-07-02T09:00:48.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 --2- 172.21.15.103:0/1440145414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f42800377c0 0x7f4280039cd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:48.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 --2- 172.21.15.103:0/1440145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4294132f50 0x7f42941ba9b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:48.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 >> 172.21.15.103:0/1440145414 conn(0x7f42940a2080 msgr2=0x7f42940a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:48.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 shutdown_connections 2023-07-02T09:00:48.895 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:48.893+0000 7f429a8df700 1 -- 172.21.15.103:0/1440145414 wait complete. 2023-07-02T09:00:48.907 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:49.908 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:50.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.047+0000 7f28149d6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:50.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.052+0000 7f28149d6700 1 Processor -- start 2023-07-02T09:00:50.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.052+0000 7f28149d6700 1 -- start start 2023-07-02T09:00:50.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f28149d6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2810132f50 0x7f2810133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:50.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f28149d6700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281009e260 con 0x7f281009c580 2023-07-02T09:00:50.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f28149d6700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28100a1640 con 0x7f2810132f50 2023-07-02T09:00:50.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f280e59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2810132f50 0x7f2810133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:50.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f2807fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f281009c580 0x7f281009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42104/0 (socket says 172.21.15.103:42104) 2023-07-02T09:00:50.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f280e59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2810132f50 0x7f2810133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34792/0 (socket says 172.21.15.103:34792) 2023-07-02T09:00:50.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f280e59c700 1 -- 172.21.15.103:0/2246629431 learned_addr learned my addr 172.21.15.103:0/2246629431 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:50.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f280e59c700 1 -- 172.21.15.103:0/2246629431 >> v1:172.21.15.103:6789/0 conn(0x7f281009c580 legacy=0x7f281009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:50.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f280e59c700 1 -- 172.21.15.103:0/2246629431 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810132a20 con 0x7f2810132f50 2023-07-02T09:00:50.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f280e59c700 1 --2- 172.21.15.103:0/2246629431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2810132f50 0x7f2810133360 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38efdb9132dc7d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:50.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.053+0000 7f280dd9b700 1 -- 172.21.15.103:0/2246629431 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2800009d90 con 0x7f2810132f50 2023-07-02T09:00:50.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f280dd9b700 1 -- 172.21.15.103:0/2246629431 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f280000e230 con 0x7f2810132f50 2023-07-02T09:00:50.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f280dd9b700 1 -- 172.21.15.103:0/2246629431 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f280000e4d0 con 0x7f2810132f50 2023-07-02T09:00:50.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f28149d6700 1 -- 172.21.15.103:0/2246629431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2810132f50 msgr2=0x7f2810133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:50.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f28149d6700 1 --2- 172.21.15.103:0/2246629431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2810132f50 0x7f2810133360 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:50.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f28149d6700 1 -- 172.21.15.103:0/2246629431 shutdown_connections 2023-07-02T09:00:50.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f28149d6700 1 --2- 172.21.15.103:0/2246629431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2810132f50 0x7f2810133360 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f28149d6700 1 -- 172.21.15.103:0/2246629431 >> 172.21.15.103:0/2246629431 conn(0x7f28100a2080 msgr2=0x7f28100a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f28149d6700 1 -- 172.21.15.103:0/2246629431 shutdown_connections 2023-07-02T09:00:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.054+0000 7f28149d6700 1 -- 172.21.15.103:0/2246629431 wait complete. 2023-07-02T09:00:50.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.055+0000 7f28149d6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:50.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.056+0000 7f28149d6700 1 Processor -- start 2023-07-02T09:00:50.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.056+0000 7f28149d6700 1 -- start start 2023-07-02T09:00:50.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.057+0000 7f28149d6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f281009c580 0x7f281012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:50.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.057+0000 7f28149d6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281006f9d0 con 0x7f281009c580 2023-07-02T09:00:50.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.057+0000 7f280e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f281009c580 0x7f281012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:50.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.057+0000 7f280e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f281009c580 0x7f281012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34794/0 (socket says 172.21.15.103:34794) 2023-07-02T09:00:50.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.057+0000 7f280e59c700 1 -- 172.21.15.103:0/3333728980 learned_addr learned my addr 172.21.15.103:0/3333728980 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:50.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.057+0000 7f280e59c700 1 -- 172.21.15.103:0/3333728980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810133a00 con 0x7f281009c580 2023-07-02T09:00:50.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.057+0000 7f280e59c700 1 --2- 172.21.15.103:0/3333728980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f281009c580 0x7f281012b7b0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:50.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.057+0000 7f2806ffd700 1 -- 172.21.15.103:0/3333728980 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2800014540 con 0x7f281009c580 2023-07-02T09:00:50.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.058+0000 7f2806ffd700 1 -- 172.21.15.103:0/3333728980 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f28000146a0 con 0x7f281009c580 2023-07-02T09:00:50.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.058+0000 7f2806ffd700 1 -- 172.21.15.103:0/3333728980 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2800014940 con 0x7f281009c580 2023-07-02T09:00:50.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.058+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281006d160 con 0x7f281009c580 2023-07-02T09:00:50.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.058+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281006da70 con 0x7f281009c580 2023-07-02T09:00:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.058+0000 7f2806ffd700 1 -- 172.21.15.103:0/3333728980 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2800014af0 con 0x7f281009c580 2023-07-02T09:00:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.059+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2810061ea0 con 0x7f281009c580 2023-07-02T09:00:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.059+0000 7f2806ffd700 1 --2- 172.21.15.103:0/3333728980 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f27f00377b0 0x7f27f0039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.059+0000 7f2806ffd700 1 -- 172.21.15.103:0/3333728980 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f28000482d0 con 0x7f281009c580 2023-07-02T09:00:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.059+0000 7f2807fff700 1 --2- 172.21.15.103:0/3333728980 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f27f00377b0 0x7f27f0039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.059+0000 7f2807fff700 1 --2- 172.21.15.103:0/3333728980 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f27f00377b0 0x7f27f0039cc0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.063+0000 7f2806ffd700 1 -- 172.21.15.103:0/3333728980 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f280000fe40 con 0x7f281009c580 2023-07-02T09:00:50.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.254+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f28100725b0 con 0x7f281009c580 2023-07-02T09:00:50.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.254+0000 7f2806ffd700 1 -- 172.21.15.103:0/3333728980 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2800010e60 con 0x7f281009c580 2023-07-02T09:00:50.255 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:50.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f27f00377b0 msgr2=0x7f27f0039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:50.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 --2- 172.21.15.103:0/3333728980 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f27f00377b0 0x7f27f0039cc0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:50.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f281009c580 msgr2=0x7f281012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:50.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 --2- 172.21.15.103:0/3333728980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f281009c580 0x7f281012b7b0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:50.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 shutdown_connections 2023-07-02T09:00:50.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 --2- 172.21.15.103:0/3333728980 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f27f00377b0 0x7f27f0039cc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:50.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 --2- 172.21.15.103:0/3333728980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f281009c580 0x7f281012b7b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:50.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 >> 172.21.15.103:0/3333728980 conn(0x7f28100a2080 msgr2=0x7f28101babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:50.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 shutdown_connections 2023-07-02T09:00:50.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:50.256+0000 7f28149d6700 1 -- 172.21.15.103:0/3333728980 wait complete. 2023-07-02T09:00:50.270 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:51.272 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:51.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.411+0000 7f2b37e49700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:51.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.416+0000 7f2b37e49700 1 Processor -- start 2023-07-02T09:00:51.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.416+0000 7f2b37e49700 1 -- start start 2023-07-02T09:00:51.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b37e49700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2b3009a220 0x7f2b301348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:51.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b37e49700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3009cea0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b37e49700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3006d490 con 0x7f2b3009a220 2023-07-02T09:00:51.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b35be5700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2b3006d6e0 0x7f2b30099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42120/0 (socket says 172.21.15.103:42120) 2023-07-02T09:00:51.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b35be5700 1 -- 172.21.15.103:0/201921166 learned_addr learned my addr 172.21.15.103:0/201921166 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:51.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b353e4700 1 --2- 172.21.15.103:0/201921166 >> v2:172.21.15.103:3300/0 conn(0x7f2b3009a220 0x7f2b301348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:51.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b353e4700 1 -- 172.21.15.103:0/201921166 >> v1:172.21.15.103:6789/0 conn(0x7f2b3006d6e0 legacy=0x7f2b30099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:51.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b34be3700 1 -- 172.21.15.103:0/201921166 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235189728 0 0) 0x7f2b3009cea0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b353e4700 1 -- 172.21.15.103:0/201921166 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3006dc80 con 0x7f2b3009a220 2023-07-02T09:00:51.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b353e4700 1 --2- 172.21.15.103:0/201921166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3009a220 0x7f2b301348e0 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cdd0ff1f713eba server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:51.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b34be3700 1 -- 172.21.15.103:0/201921166 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2b20009cd0 con 0x7f2b3009a220 2023-07-02T09:00:51.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b34be3700 1 -- 172.21.15.103:0/201921166 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2b20009e30 con 0x7f2b3009a220 2023-07-02T09:00:51.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.417+0000 7f2b34be3700 1 -- 172.21.15.103:0/201921166 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2b2000e360 con 0x7f2b3009a220 2023-07-02T09:00:51.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.418+0000 7f2b37e49700 1 -- 172.21.15.103:0/201921166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3009a220 msgr2=0x7f2b301348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:51.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.418+0000 7f2b37e49700 1 --2- 172.21.15.103:0/201921166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3009a220 0x7f2b301348e0 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:51.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.418+0000 7f2b37e49700 1 -- 172.21.15.103:0/201921166 shutdown_connections 2023-07-02T09:00:51.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.418+0000 7f2b37e49700 1 --2- 172.21.15.103:0/201921166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3009a220 0x7f2b301348e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:51.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.418+0000 7f2b37e49700 1 -- 172.21.15.103:0/201921166 >> 172.21.15.103:0/201921166 conn(0x7f2b3009d810 msgr2=0x7f2b3009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:51.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.418+0000 7f2b37e49700 1 -- 172.21.15.103:0/201921166 shutdown_connections 2023-07-02T09:00:51.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.418+0000 7f2b37e49700 1 -- 172.21.15.103:0/201921166 wait complete. 2023-07-02T09:00:51.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.419+0000 7f2b37e49700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:51.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.419+0000 7f2b37e49700 1 Processor -- start 2023-07-02T09:00:51.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.420+0000 7f2b37e49700 1 -- start start 2023-07-02T09:00:51.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.420+0000 7f2b37e49700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3006d6e0 0x7f2b301201a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:51.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.420+0000 7f2b37e49700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b30122e00 con 0x7f2b3006d6e0 2023-07-02T09:00:51.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.420+0000 7f2b35be5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3006d6e0 0x7f2b301201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:51.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.420+0000 7f2b35be5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3006d6e0 0x7f2b301201a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34810/0 (socket says 172.21.15.103:34810) 2023-07-02T09:00:51.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.420+0000 7f2b35be5700 1 -- 172.21.15.103:0/4016991864 learned_addr learned my addr 172.21.15.103:0/4016991864 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:51.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.421+0000 7f2b35be5700 1 -- 172.21.15.103:0/4016991864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3011fd70 con 0x7f2b3006d6e0 2023-07-02T09:00:51.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.421+0000 7f2b35be5700 1 --2- 172.21.15.103:0/4016991864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3006d6e0 0x7f2b301201a0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:51.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.421+0000 7f2b267fc700 1 -- 172.21.15.103:0/4016991864 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2b2c004b80 con 0x7f2b3006d6e0 2023-07-02T09:00:51.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.421+0000 7f2b267fc700 1 -- 172.21.15.103:0/4016991864 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2b2c004ce0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.421+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b30121fc0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.421+0000 7f2b267fc700 1 -- 172.21.15.103:0/4016991864 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2b2c00c550 con 0x7f2b3006d6e0 2023-07-02T09:00:51.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.421+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b301228c0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.422+0000 7f2b267fc700 1 -- 172.21.15.103:0/4016991864 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2b2c00c6b0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.422+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b30061ea0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.422+0000 7f2b267fc700 1 --2- 172.21.15.103:0/4016991864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2b140377c0 0x7f2b14039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:51.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.422+0000 7f2b267fc700 1 -- 172.21.15.103:0/4016991864 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2b2c046ae0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.422+0000 7f2b353e4700 1 --2- 172.21.15.103:0/4016991864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2b140377c0 0x7f2b14039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:51.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.423+0000 7f2b353e4700 1 --2- 172.21.15.103:0/4016991864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2b140377c0 0x7f2b14039cd0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:51.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.427+0000 7f2b267fc700 1 -- 172.21.15.103:0/4016991864 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2b2c009a80 con 0x7f2b3006d6e0 2023-07-02T09:00:51.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.615+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2b30122150 con 0x7f2b3006d6e0 2023-07-02T09:00:51.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.616+0000 7f2b267fc700 1 -- 172.21.15.103:0/4016991864 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2b2c0098a0 con 0x7f2b3006d6e0 2023-07-02T09:00:51.617 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:51.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2b140377c0 msgr2=0x7f2b14039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:51.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 --2- 172.21.15.103:0/4016991864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2b140377c0 0x7f2b14039cd0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:51.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3006d6e0 msgr2=0x7f2b301201a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:51.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 --2- 172.21.15.103:0/4016991864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3006d6e0 0x7f2b301201a0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:51.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 shutdown_connections 2023-07-02T09:00:51.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 --2- 172.21.15.103:0/4016991864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2b140377c0 0x7f2b14039cd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:51.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 --2- 172.21.15.103:0/4016991864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3006d6e0 0x7f2b301201a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:51.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 >> 172.21.15.103:0/4016991864 conn(0x7f2b3009d810 msgr2=0x7f2b30117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:51.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 shutdown_connections 2023-07-02T09:00:51.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:51.618+0000 7f2b37e49700 1 -- 172.21.15.103:0/4016991864 wait complete. 2023-07-02T09:00:51.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:51.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~66s 2023-07-02T09:00:52.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:52.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.773+0000 7fd0c27c9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:52.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.777+0000 7fd0c27c9700 1 Processor -- start 2023-07-02T09:00:52.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.777+0000 7fd0c27c9700 1 -- start start 2023-07-02T09:00:52.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0c27c9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd0bc132030 0x7fd0bc134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:52.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0c27c9700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc06df60 con 0x7fd0bc12f250 2023-07-02T09:00:52.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0c27c9700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc09a710 con 0x7fd0bc132030 2023-07-02T09:00:52.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0bb7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd0bc132030 0x7fd0bc134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:52.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0bbfff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd0bc12f250 0x7fd0bc12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42126/0 (socket says 172.21.15.103:42126) 2023-07-02T09:00:52.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0bb7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd0bc132030 0x7fd0bc134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34812/0 (socket says 172.21.15.103:34812) 2023-07-02T09:00:52.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0bbfff700 1 -- 172.21.15.103:0/668782152 learned_addr learned my addr 172.21.15.103:0/668782152 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:52.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0bb7fe700 1 -- 172.21.15.103:0/668782152 >> v1:172.21.15.103:6789/0 conn(0x7fd0bc12f250 legacy=0x7fd0bc12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:52.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0bb7fe700 1 -- 172.21.15.103:0/668782152 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc12ed20 con 0x7fd0bc132030 2023-07-02T09:00:52.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0baffd700 1 -- 172.21.15.103:0/668782152 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060445905 0 0) 0x7fd0bc06df60 con 0x7fd0bc12f250 2023-07-02T09:00:52.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0bb7fe700 1 --2- 172.21.15.103:0/668782152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc132030 0x7fd0bc134450 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88163570b3c65b27 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:52.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0baffd700 1 -- 172.21.15.103:0/668782152 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd0ac00d450 con 0x7fd0bc132030 2023-07-02T09:00:52.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0baffd700 1 -- 172.21.15.103:0/668782152 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd0ac00d5b0 con 0x7fd0bc132030 2023-07-02T09:00:52.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.778+0000 7fd0baffd700 1 -- 172.21.15.103:0/668782152 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd0ac00d850 con 0x7fd0bc132030 2023-07-02T09:00:52.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.779+0000 7fd0c27c9700 1 -- 172.21.15.103:0/668782152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc132030 msgr2=0x7fd0bc134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:52.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.779+0000 7fd0c27c9700 1 --2- 172.21.15.103:0/668782152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc132030 0x7fd0bc134450 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:52.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.779+0000 7fd0c27c9700 1 -- 172.21.15.103:0/668782152 shutdown_connections 2023-07-02T09:00:52.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.779+0000 7fd0c27c9700 1 --2- 172.21.15.103:0/668782152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc132030 0x7fd0bc134450 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:52.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.779+0000 7fd0c27c9700 1 -- 172.21.15.103:0/668782152 >> 172.21.15.103:0/668782152 conn(0x7fd0bc09d810 msgr2=0x7fd0bc09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:52.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.779+0000 7fd0c27c9700 1 -- 172.21.15.103:0/668782152 shutdown_connections 2023-07-02T09:00:52.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.779+0000 7fd0c27c9700 1 -- 172.21.15.103:0/668782152 wait complete. 2023-07-02T09:00:52.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.780+0000 7fd0c27c9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:52.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.781+0000 7fd0c27c9700 1 Processor -- start 2023-07-02T09:00:52.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.781+0000 7fd0c27c9700 1 -- start start 2023-07-02T09:00:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0c27c9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc12f250 0x7fd0bc06fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0c27c9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc071390 con 0x7fd0bc12f250 2023-07-02T09:00:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0bbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc12f250 0x7fd0bc06fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0bbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc12f250 0x7fd0bc06fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34818/0 (socket says 172.21.15.103:34818) 2023-07-02T09:00:52.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0bbfff700 1 -- 172.21.15.103:0/3491016768 learned_addr learned my addr 172.21.15.103:0/3491016768 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:52.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0bbfff700 1 -- 172.21.15.103:0/3491016768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc1292e0 con 0x7fd0bc12f250 2023-07-02T09:00:52.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0bbfff700 1 --2- 172.21.15.103:0/3491016768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc12f250 0x7fd0bc06fe70 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:52.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0b8ff9700 1 -- 172.21.15.103:0/3491016768 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd0b0004b80 con 0x7fd0bc12f250 2023-07-02T09:00:52.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0b8ff9700 1 -- 172.21.15.103:0/3491016768 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd0b0004ce0 con 0x7fd0bc12f250 2023-07-02T09:00:52.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0b8ff9700 1 -- 172.21.15.103:0/3491016768 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd0b000c4f0 con 0x7fd0bc12f250 2023-07-02T09:00:52.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc128f40 con 0x7fd0bc12f250 2023-07-02T09:00:52.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.782+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc1179c0 con 0x7fd0bc12f250 2023-07-02T09:00:52.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.783+0000 7fd0b8ff9700 1 -- 172.21.15.103:0/3491016768 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd0b000c650 con 0x7fd0bc12f250 2023-07-02T09:00:52.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.783+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0bc061ea0 con 0x7fd0bc12f250 2023-07-02T09:00:52.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.783+0000 7fd0b8ff9700 1 --2- 172.21.15.103:0/3491016768 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0a80377c0 0x7fd0a8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:52.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.783+0000 7fd0b8ff9700 1 -- 172.21.15.103:0/3491016768 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd0b00467f0 con 0x7fd0bc12f250 2023-07-02T09:00:52.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.784+0000 7fd0bb7fe700 1 --2- 172.21.15.103:0/3491016768 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0a80377c0 0x7fd0a8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.784+0000 7fd0bb7fe700 1 --2- 172.21.15.103:0/3491016768 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0a80377c0 0x7fd0a8039cd0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.789+0000 7fd0b8ff9700 1 -- 172.21.15.103:0/3491016768 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd0b0009ae0 con 0x7fd0bc12f250 2023-07-02T09:00:52.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.981+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd0bc1292c0 con 0x7fd0bc12f250 2023-07-02T09:00:52.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.981+0000 7fd0b8ff9700 1 -- 172.21.15.103:0/3491016768 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd0b00090b0 con 0x7fd0bc12f250 2023-07-02T09:00:52.983 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:52.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.983+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0a80377c0 msgr2=0x7fd0a8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:52.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.983+0000 7fd0c27c9700 1 --2- 172.21.15.103:0/3491016768 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0a80377c0 0x7fd0a8039cd0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:52.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.983+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc12f250 msgr2=0x7fd0bc06fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:52.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.983+0000 7fd0c27c9700 1 --2- 172.21.15.103:0/3491016768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc12f250 0x7fd0bc06fe70 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:52.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.983+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 shutdown_connections 2023-07-02T09:00:52.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.983+0000 7fd0c27c9700 1 --2- 172.21.15.103:0/3491016768 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0a80377c0 0x7fd0a8039cd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:52.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.983+0000 7fd0c27c9700 1 --2- 172.21.15.103:0/3491016768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0bc12f250 0x7fd0bc06fe70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:52.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.983+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 >> 172.21.15.103:0/3491016768 conn(0x7fd0bc09d810 msgr2=0x7fd0bc12dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:52.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.984+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 shutdown_connections 2023-07-02T09:00:52.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:52.984+0000 7fd0c27c9700 1 -- 172.21.15.103:0/3491016768 wait complete. 2023-07-02T09:00:52.998 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:53.998 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:54.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.136+0000 7f11dac62700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:54.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.141+0000 7f11dac62700 1 Processor -- start 2023-07-02T09:00:54.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.142+0000 7f11dac62700 1 -- start start 2023-07-02T09:00:54.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11dac62700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11d412f250 0x7f11d412f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:54.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11dac62700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d406df60 con 0x7f11d4098a00 2023-07-02T09:00:54.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11dac62700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d409a710 con 0x7f11d412f250 2023-07-02T09:00:54.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d89fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11d412f250 0x7f11d412f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:54.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d3fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f11d4098a00 0x7f11d4098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57028/0 (socket says 172.21.15.103:57028) 2023-07-02T09:00:54.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d89fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11d412f250 0x7f11d412f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54636/0 (socket says 172.21.15.103:54636) 2023-07-02T09:00:54.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d3fff700 1 -- 172.21.15.103:0/2562891735 learned_addr learned my addr 172.21.15.103:0/2562891735 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d89fe700 1 -- 172.21.15.103:0/2562891735 >> v1:172.21.15.103:6789/0 conn(0x7f11d4098a00 legacy=0x7f11d4098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d37fe700 1 -- 172.21.15.103:0/2562891735 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453547339 0 0) 0x7f11d406df60 con 0x7f11d4098a00 2023-07-02T09:00:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d89fe700 1 -- 172.21.15.103:0/2562891735 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d412ed20 con 0x7f11d412f250 2023-07-02T09:00:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d89fe700 1 --2- 172.21.15.103:0/2562891735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d412f250 0x7f11d412f660 crc :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31728a3ee7dae93b server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:54.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d37fe700 1 -- 172.21.15.103:0/2562891735 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f11c4009cd0 con 0x7f11d412f250 2023-07-02T09:00:54.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d37fe700 1 -- 172.21.15.103:0/2562891735 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f11c4009e30 con 0x7f11d412f250 2023-07-02T09:00:54.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.143+0000 7f11d37fe700 1 -- 172.21.15.103:0/2562891735 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f11c400e360 con 0x7f11d412f250 2023-07-02T09:00:54.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.144+0000 7f11dac62700 1 -- 172.21.15.103:0/2562891735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d412f250 msgr2=0x7f11d412f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:54.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.144+0000 7f11dac62700 1 --2- 172.21.15.103:0/2562891735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d412f250 0x7f11d412f660 crc :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:54.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.144+0000 7f11dac62700 1 -- 172.21.15.103:0/2562891735 shutdown_connections 2023-07-02T09:00:54.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.144+0000 7f11dac62700 1 --2- 172.21.15.103:0/2562891735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d412f250 0x7f11d412f660 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:54.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.144+0000 7f11dac62700 1 -- 172.21.15.103:0/2562891735 >> 172.21.15.103:0/2562891735 conn(0x7f11d409d810 msgr2=0x7f11d409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:54.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.144+0000 7f11dac62700 1 -- 172.21.15.103:0/2562891735 shutdown_connections 2023-07-02T09:00:54.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.144+0000 7f11dac62700 1 -- 172.21.15.103:0/2562891735 wait complete. 2023-07-02T09:00:54.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.144+0000 7f11dac62700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:54.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.145+0000 7f11dac62700 1 Processor -- start 2023-07-02T09:00:54.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.145+0000 7f11dac62700 1 -- start start 2023-07-02T09:00:54.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.146+0000 7f11dac62700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d4098a00 0x7f11d4117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:54.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.146+0000 7f11dac62700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4118750 con 0x7f11d4098a00 2023-07-02T09:00:54.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.146+0000 7f11d89fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d4098a00 0x7f11d4117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:54.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.146+0000 7f11d89fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d4098a00 0x7f11d4117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54652/0 (socket says 172.21.15.103:54652) 2023-07-02T09:00:54.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.146+0000 7f11d89fe700 1 -- 172.21.15.103:0/4259941762 learned_addr learned my addr 172.21.15.103:0/4259941762 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:54.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.147+0000 7f11d89fe700 1 -- 172.21.15.103:0/4259941762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d411a450 con 0x7f11d4098a00 2023-07-02T09:00:54.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.147+0000 7f11d89fe700 1 --2- 172.21.15.103:0/4259941762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d4098a00 0x7f11d4117360 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:54.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.147+0000 7f11d17fa700 1 -- 172.21.15.103:0/4259941762 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f11c4013530 con 0x7f11d4098a00 2023-07-02T09:00:54.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.147+0000 7f11d17fa700 1 -- 172.21.15.103:0/4259941762 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f11c4013690 con 0x7f11d4098a00 2023-07-02T09:00:54.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.147+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d4118e20 con 0x7f11d4098a00 2023-07-02T09:00:54.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.147+0000 7f11d17fa700 1 -- 172.21.15.103:0/4259941762 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f11c4013930 con 0x7f11d4098a00 2023-07-02T09:00:54.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.147+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d4119e70 con 0x7f11d4098a00 2023-07-02T09:00:54.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.148+0000 7f11d17fa700 1 -- 172.21.15.103:0/4259941762 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f11c4013ae0 con 0x7f11d4098a00 2023-07-02T09:00:54.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.148+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d4061ea0 con 0x7f11d4098a00 2023-07-02T09:00:54.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.148+0000 7f11d17fa700 1 --2- 172.21.15.103:0/4259941762 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377b0 0x7f11c0039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:54.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.148+0000 7f11d17fa700 1 -- 172.21.15.103:0/4259941762 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f11c4018040 con 0x7f11d4098a00 2023-07-02T09:00:54.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.148+0000 7f11d3fff700 1 --2- 172.21.15.103:0/4259941762 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377b0 0x7f11c0039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:54.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.149+0000 7f11d3fff700 1 --2- 172.21.15.103:0/4259941762 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377b0 0x7f11c0039cc0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:54.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.153+0000 7f11d17fa700 1 -- 172.21.15.103:0/4259941762 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f11c4018e00 con 0x7f11d4098a00 2023-07-02T09:00:54.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.344+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f11d4073ca0 con 0x7f11d4098a00 2023-07-02T09:00:54.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.345+0000 7f11d17fa700 1 -- 172.21.15.103:0/4259941762 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f11c4010e60 con 0x7f11d4098a00 2023-07-02T09:00:54.345 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:54.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377b0 msgr2=0x7f11c0039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:54.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 --2- 172.21.15.103:0/4259941762 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377b0 0x7f11c0039cc0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:54.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d4098a00 msgr2=0x7f11d4117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:54.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 --2- 172.21.15.103:0/4259941762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d4098a00 0x7f11d4117360 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:54.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 shutdown_connections 2023-07-02T09:00:54.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 --2- 172.21.15.103:0/4259941762 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377b0 0x7f11c0039cc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:54.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 --2- 172.21.15.103:0/4259941762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11d4098a00 0x7f11d4117360 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:54.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 >> 172.21.15.103:0/4259941762 conn(0x7f11d409d810 msgr2=0x7f11d40a0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:54.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 shutdown_connections 2023-07-02T09:00:54.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:54.347+0000 7f11dac62700 1 -- 172.21.15.103:0/4259941762 wait complete. 2023-07-02T09:00:54.360 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:55.361 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.500+0000 7f22c29a9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:55.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.505+0000 7f22c29a9700 1 Processor -- start 2023-07-02T09:00:55.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.505+0000 7f22c29a9700 1 -- start start 2023-07-02T09:00:55.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22c29a9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f22bc1331e0 0x7f22bc13c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:55.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22c29a9700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22bc09aff0 con 0x7f22bc09aa70 2023-07-02T09:00:55.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22c29a9700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc09a860 con 0x7f22bc1331e0 2023-07-02T09:00:55.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22bb7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f22bc1331e0 0x7f22bc13c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:55.507 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22bbfff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f22bc09aa70 0x7f22bc132ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57032/0 (socket says 172.21.15.103:57032) 2023-07-02T09:00:55.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22bb7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f22bc1331e0 0x7f22bc13c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54666/0 (socket says 172.21.15.103:54666) 2023-07-02T09:00:55.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22bbfff700 1 -- 172.21.15.103:0/3831325083 learned_addr learned my addr 172.21.15.103:0/3831325083 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:55.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22bb7fe700 1 -- 172.21.15.103:0/3831325083 >> v1:172.21.15.103:6789/0 conn(0x7f22bc09aa70 legacy=0x7f22bc132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:55.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22bb7fe700 1 -- 172.21.15.103:0/3831325083 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc06d6a0 con 0x7f22bc1331e0 2023-07-02T09:00:55.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22baffd700 1 -- 172.21.15.103:0/3831325083 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881194072 0 0) 0x7f22bc09aff0 con 0x7f22bc09aa70 2023-07-02T09:00:55.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22bb7fe700 1 --2- 172.21.15.103:0/3831325083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc1331e0 0x7f22bc13c460 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e151ff9617a47e8 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:55.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22baffd700 1 -- 172.21.15.103:0/3831325083 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f22ac01ad70 con 0x7f22bc1331e0 2023-07-02T09:00:55.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22baffd700 1 -- 172.21.15.103:0/3831325083 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f22ac01f240 con 0x7f22bc1331e0 2023-07-02T09:00:55.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.506+0000 7f22baffd700 1 -- 172.21.15.103:0/3831325083 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f22ac01f3b0 con 0x7f22bc1331e0 2023-07-02T09:00:55.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.507+0000 7f22c29a9700 1 -- 172.21.15.103:0/3831325083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc1331e0 msgr2=0x7f22bc13c460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:55.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.507+0000 7f22c29a9700 1 --2- 172.21.15.103:0/3831325083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc1331e0 0x7f22bc13c460 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:55.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.507+0000 7f22c29a9700 1 -- 172.21.15.103:0/3831325083 shutdown_connections 2023-07-02T09:00:55.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.507+0000 7f22c29a9700 1 --2- 172.21.15.103:0/3831325083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc1331e0 0x7f22bc13c460 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:55.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.507+0000 7f22c29a9700 1 -- 172.21.15.103:0/3831325083 >> 172.21.15.103:0/3831325083 conn(0x7f22bc09d810 msgr2=0x7f22bc09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:55.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.507+0000 7f22c29a9700 1 -- 172.21.15.103:0/3831325083 shutdown_connections 2023-07-02T09:00:55.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.507+0000 7f22c29a9700 1 -- 172.21.15.103:0/3831325083 wait complete. 2023-07-02T09:00:55.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.507+0000 7f22c29a9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:55.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.508+0000 7f22c29a9700 1 Processor -- start 2023-07-02T09:00:55.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.508+0000 7f22c29a9700 1 -- start start 2023-07-02T09:00:55.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.509+0000 7f22c29a9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc09aa70 0x7f22bc1c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:55.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.509+0000 7f22c29a9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc1b8960 con 0x7f22bc09aa70 2023-07-02T09:00:55.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.509+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc09aa70 0x7f22bc1c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:55.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.509+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc09aa70 0x7f22bc1c2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54682/0 (socket says 172.21.15.103:54682) 2023-07-02T09:00:55.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.509+0000 7f22bbfff700 1 -- 172.21.15.103:0/1041293711 learned_addr learned my addr 172.21.15.103:0/1041293711 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:55.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.509+0000 7f22bbfff700 1 -- 172.21.15.103:0/1041293711 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc11f050 con 0x7f22bc09aa70 2023-07-02T09:00:55.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.510+0000 7f22bbfff700 1 --2- 172.21.15.103:0/1041293711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc09aa70 0x7f22bc1c2d90 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:55.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.510+0000 7f22b8ff9700 1 -- 172.21.15.103:0/1041293711 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f22b0004b80 con 0x7f22bc09aa70 2023-07-02T09:00:55.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.510+0000 7f22b8ff9700 1 -- 172.21.15.103:0/1041293711 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f22b0004ce0 con 0x7f22bc09aa70 2023-07-02T09:00:55.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.510+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc132650 con 0x7f22bc09aa70 2023-07-02T09:00:55.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.510+0000 7f22b8ff9700 1 -- 172.21.15.103:0/1041293711 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f22b000c590 con 0x7f22bc09aa70 2023-07-02T09:00:55.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.510+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc11e700 con 0x7f22bc09aa70 2023-07-02T09:00:55.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.510+0000 7f22b8ff9700 1 -- 172.21.15.103:0/1041293711 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f22b000c740 con 0x7f22bc09aa70 2023-07-02T09:00:55.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.511+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc1324c0 con 0x7f22bc09aa70 2023-07-02T09:00:55.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.511+0000 7f22b8ff9700 1 --2- 172.21.15.103:0/1041293711 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22a80377c0 0x7f22a8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:55.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.511+0000 7f22b8ff9700 1 -- 172.21.15.103:0/1041293711 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f22b0046a30 con 0x7f22bc09aa70 2023-07-02T09:00:55.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.511+0000 7f22bb7fe700 1 --2- 172.21.15.103:0/1041293711 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22a80377c0 0x7f22a8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:55.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.512+0000 7f22bb7fe700 1 --2- 172.21.15.103:0/1041293711 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22a80377c0 0x7f22a8039cd0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:55.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.515+0000 7f22b8ff9700 1 -- 172.21.15.103:0/1041293711 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f22b0009c50 con 0x7f22bc09aa70 2023-07-02T09:00:55.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.703+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f22bc061ea0 con 0x7f22bc09aa70 2023-07-02T09:00:55.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.704+0000 7f22b8ff9700 1 -- 172.21.15.103:0/1041293711 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f22b0009c50 con 0x7f22bc09aa70 2023-07-02T09:00:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:55.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22a80377c0 msgr2=0x7f22a8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:55.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 --2- 172.21.15.103:0/1041293711 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22a80377c0 0x7f22a8039cd0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:55.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc09aa70 msgr2=0x7f22bc1c2d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:55.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 --2- 172.21.15.103:0/1041293711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc09aa70 0x7f22bc1c2d90 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:55.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 shutdown_connections 2023-07-02T09:00:55.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 --2- 172.21.15.103:0/1041293711 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22a80377c0 0x7f22a8039cd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:55.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 --2- 172.21.15.103:0/1041293711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22bc09aa70 0x7f22bc1c2d90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:55.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 >> 172.21.15.103:0/1041293711 conn(0x7f22bc09d810 msgr2=0x7f22bc119610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:55.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 shutdown_connections 2023-07-02T09:00:55.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:55.706+0000 7f22c29a9700 1 -- 172.21.15.103:0/1041293711 wait complete. 2023-07-02T09:00:55.718 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:56.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:56.860 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.856+0000 7f301a6dd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.861+0000 7f301a6dd700 1 Processor -- start 2023-07-02T09:00:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.861+0000 7f301a6dd700 1 -- start start 2023-07-02T09:00:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f301a6dd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3014132030 0x7f3014134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f301a6dd700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301406df60 con 0x7f301412f250 2023-07-02T09:00:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f301a6dd700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f301409a710 con 0x7f3014132030 2023-07-02T09:00:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f30137fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3014132030 0x7f3014134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f3013fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f301412f250 0x7f301412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57044/0 (socket says 172.21.15.103:57044) 2023-07-02T09:00:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f3013fff700 1 -- 172.21.15.103:0/778061000 learned_addr learned my addr 172.21.15.103:0/778061000 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f30137fe700 1 -- 172.21.15.103:0/778061000 >> v1:172.21.15.103:6789/0 conn(0x7f301412f250 legacy=0x7f301412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:56.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f30137fe700 1 -- 172.21.15.103:0/778061000 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301412ed20 con 0x7f3014132030 2023-07-02T09:00:56.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f30137fe700 1 --2- 172.21.15.103:0/778061000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014132030 0x7f3014134450 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1f0d2db0bc2bff server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:56.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f3012ffd700 1 -- 172.21.15.103:0/778061000 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3004009cd0 con 0x7f3014132030 2023-07-02T09:00:56.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f3012ffd700 1 -- 172.21.15.103:0/778061000 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3004009e30 con 0x7f3014132030 2023-07-02T09:00:56.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.862+0000 7f3012ffd700 1 -- 172.21.15.103:0/778061000 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f300400e360 con 0x7f3014132030 2023-07-02T09:00:56.865 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.863+0000 7f301a6dd700 1 -- 172.21.15.103:0/778061000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014132030 msgr2=0x7f3014134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:56.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.863+0000 7f301a6dd700 1 --2- 172.21.15.103:0/778061000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014132030 0x7f3014134450 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:56.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.863+0000 7f301a6dd700 1 -- 172.21.15.103:0/778061000 shutdown_connections 2023-07-02T09:00:56.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.863+0000 7f301a6dd700 1 --2- 172.21.15.103:0/778061000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014132030 0x7f3014134450 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:56.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.863+0000 7f301a6dd700 1 -- 172.21.15.103:0/778061000 >> 172.21.15.103:0/778061000 conn(0x7f301409d810 msgr2=0x7f301409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:56.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.863+0000 7f301a6dd700 1 -- 172.21.15.103:0/778061000 shutdown_connections 2023-07-02T09:00:56.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.863+0000 7f301a6dd700 1 -- 172.21.15.103:0/778061000 wait complete. 2023-07-02T09:00:56.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.864+0000 7f301a6dd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:56.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.864+0000 7f301a6dd700 1 Processor -- start 2023-07-02T09:00:56.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.865+0000 7f301a6dd700 1 -- start start 2023-07-02T09:00:56.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.865+0000 7f301a6dd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f301412f250 0x7f30141c5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:56.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.865+0000 7f301a6dd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301411a080 con 0x7f301412f250 2023-07-02T09:00:56.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.865+0000 7f3013fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f301412f250 0x7f30141c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:56.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f3013fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f301412f250 0x7f30141c5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54694/0 (socket says 172.21.15.103:54694) 2023-07-02T09:00:56.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f3013fff700 1 -- 172.21.15.103:0/1748831529 learned_addr learned my addr 172.21.15.103:0/1748831529 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:56.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f3013fff700 1 -- 172.21.15.103:0/1748831529 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30141c4ce0 con 0x7f301412f250 2023-07-02T09:00:56.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f3013fff700 1 --2- 172.21.15.103:0/1748831529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f301412f250 0x7f30141c5110 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:56.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f3010ff9700 1 -- 172.21.15.103:0/1748831529 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3008005a60 con 0x7f301412f250 2023-07-02T09:00:56.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f3010ff9700 1 -- 172.21.15.103:0/1748831529 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3008005bc0 con 0x7f301412f250 2023-07-02T09:00:56.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3014118450 con 0x7f301412f250 2023-07-02T09:00:56.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f3010ff9700 1 -- 172.21.15.103:0/1748831529 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3008005e60 con 0x7f301412f250 2023-07-02T09:00:56.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.866+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3014118890 con 0x7f301412f250 2023-07-02T09:00:56.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.867+0000 7f3010ff9700 1 -- 172.21.15.103:0/1748831529 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f30080070b0 con 0x7f301412f250 2023-07-02T09:00:56.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.867+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3014061ea0 con 0x7f301412f250 2023-07-02T09:00:56.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.867+0000 7f3010ff9700 1 --2- 172.21.15.103:0/1748831529 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30000377c0 0x7f3000039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:56.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.867+0000 7f3010ff9700 1 -- 172.21.15.103:0/1748831529 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3008046b20 con 0x7f301412f250 2023-07-02T09:00:56.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.867+0000 7f30137fe700 1 --2- 172.21.15.103:0/1748831529 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30000377c0 0x7f3000039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:56.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.868+0000 7f30137fe700 1 --2- 172.21.15.103:0/1748831529 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30000377c0 0x7f3000039cd0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:56.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:56.872+0000 7f3010ff9700 1 -- 172.21.15.103:0/1748831529 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f300800fdf0 con 0x7f301412f250 2023-07-02T09:00:57.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.059+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3014073ca0 con 0x7f301412f250 2023-07-02T09:00:57.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.059+0000 7f3010ff9700 1 -- 172.21.15.103:0/1748831529 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f300800ddd0 con 0x7f301412f250 2023-07-02T09:00:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:57.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30000377c0 msgr2=0x7f3000039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:57.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 --2- 172.21.15.103:0/1748831529 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30000377c0 0x7f3000039cd0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:57.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f301412f250 msgr2=0x7f30141c5110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:57.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 --2- 172.21.15.103:0/1748831529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f301412f250 0x7f30141c5110 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:57.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 shutdown_connections 2023-07-02T09:00:57.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 --2- 172.21.15.103:0/1748831529 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30000377c0 0x7f3000039cd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:57.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 --2- 172.21.15.103:0/1748831529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f301412f250 0x7f30141c5110 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:57.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 >> 172.21.15.103:0/1748831529 conn(0x7f301409d810 msgr2=0x7f301412e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:57.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 shutdown_connections 2023-07-02T09:00:57.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:57.062+0000 7f301a6dd700 1 -- 172.21.15.103:0/1748831529 wait complete. 2023-07-02T09:00:57.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:57.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~72s 2023-07-02T09:00:58.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:58.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.208+0000 7f84a1c1e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:58.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.213+0000 7f84a1c1e700 1 Processor -- start 2023-07-02T09:00:58.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.213+0000 7f84a1c1e700 1 -- start start 2023-07-02T09:00:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f84a1c1e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f849c133360 0x7f849c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f84a1c1e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c09e260 con 0x7f849c132f50 2023-07-02T09:00:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f84a1c1e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849c0a1640 con 0x7f849c133360 2023-07-02T09:00:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f849affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f849c133360 0x7f849c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f849b7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f849c132f50 0x7f849c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57052/0 (socket says 172.21.15.103:57052) 2023-07-02T09:00:58.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f849affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f849c133360 0x7f849c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54698/0 (socket says 172.21.15.103:54698) 2023-07-02T09:00:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f849b7fe700 1 -- 172.21.15.103:0/1378688132 learned_addr learned my addr 172.21.15.103:0/1378688132 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f849affd700 1 -- 172.21.15.103:0/1378688132 >> v1:172.21.15.103:6789/0 conn(0x7f849c132f50 legacy=0x7f849c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f849affd700 1 -- 172.21.15.103:0/1378688132 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c132a20 con 0x7f849c133360 2023-07-02T09:00:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.214+0000 7f849a7fc700 1 -- 172.21.15.103:0/1378688132 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317864336 0 0) 0x7f849c09e260 con 0x7f849c132f50 2023-07-02T09:00:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f849affd700 1 --2- 172.21.15.103:0/1378688132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c133360 0x7f849c09cc90 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404400a4dffda591 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f849a7fc700 1 -- 172.21.15.103:0/1378688132 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f848c009cd0 con 0x7f849c133360 2023-07-02T09:00:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f849a7fc700 1 -- 172.21.15.103:0/1378688132 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f848c009e30 con 0x7f849c133360 2023-07-02T09:00:58.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f849a7fc700 1 -- 172.21.15.103:0/1378688132 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f848c00e360 con 0x7f849c133360 2023-07-02T09:00:58.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f84a1c1e700 1 -- 172.21.15.103:0/1378688132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c133360 msgr2=0x7f849c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:58.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f84a1c1e700 1 --2- 172.21.15.103:0/1378688132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c133360 0x7f849c09cc90 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:58.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f84a1c1e700 1 -- 172.21.15.103:0/1378688132 shutdown_connections 2023-07-02T09:00:58.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f84a1c1e700 1 --2- 172.21.15.103:0/1378688132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c133360 0x7f849c09cc90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:58.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f84a1c1e700 1 -- 172.21.15.103:0/1378688132 >> 172.21.15.103:0/1378688132 conn(0x7f849c0a2080 msgr2=0x7f849c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:58.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f84a1c1e700 1 -- 172.21.15.103:0/1378688132 shutdown_connections 2023-07-02T09:00:58.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.215+0000 7f84a1c1e700 1 -- 172.21.15.103:0/1378688132 wait complete. 2023-07-02T09:00:58.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.216+0000 7f84a1c1e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:58.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.216+0000 7f84a1c1e700 1 Processor -- start 2023-07-02T09:00:58.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.217+0000 7f84a1c1e700 1 -- start start 2023-07-02T09:00:58.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.217+0000 7f84a1c1e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c132f50 0x7f849c1ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:58.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.217+0000 7f84a1c1e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c06d370 con 0x7f849c132f50 2023-07-02T09:00:58.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.217+0000 7f849b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c132f50 0x7f849c1ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:58.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f849b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c132f50 0x7f849c1ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54712/0 (socket says 172.21.15.103:54712) 2023-07-02T09:00:58.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f849b7fe700 1 -- 172.21.15.103:0/4073972430 learned_addr learned my addr 172.21.15.103:0/4073972430 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:58.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f849b7fe700 1 -- 172.21.15.103:0/4073972430 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c06dde0 con 0x7f849c132f50 2023-07-02T09:00:58.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f849b7fe700 1 --2- 172.21.15.103:0/4073972430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c132f50 0x7f849c1ba9b0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:58.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f8493fff700 1 -- 172.21.15.103:0/4073972430 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8484004b80 con 0x7f849c132f50 2023-07-02T09:00:58.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f8493fff700 1 -- 172.21.15.103:0/4073972430 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8484004ce0 con 0x7f849c132f50 2023-07-02T09:00:58.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849c1ba4f0 con 0x7f849c132f50 2023-07-02T09:00:58.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f8493fff700 1 -- 172.21.15.103:0/4073972430 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f848400c4f0 con 0x7f849c132f50 2023-07-02T09:00:58.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.218+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c071e40 con 0x7f849c132f50 2023-07-02T09:00:58.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.219+0000 7f8493fff700 1 -- 172.21.15.103:0/4073972430 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f848400c650 con 0x7f849c132f50 2023-07-02T09:00:58.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.219+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c061ea0 con 0x7f849c132f50 2023-07-02T09:00:58.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.219+0000 7f8493fff700 1 --2- 172.21.15.103:0/4073972430 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f847c0377c0 0x7f847c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:58.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.219+0000 7f8493fff700 1 -- 172.21.15.103:0/4073972430 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f8484046970 con 0x7f849c132f50 2023-07-02T09:00:58.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.219+0000 7f849affd700 1 --2- 172.21.15.103:0/4073972430 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f847c0377c0 0x7f847c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:58.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.220+0000 7f849affd700 1 --2- 172.21.15.103:0/4073972430 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f847c0377c0 0x7f847c039cd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:58.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.224+0000 7f8493fff700 1 -- 172.21.15.103:0/4073972430 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f848400e540 con 0x7f849c132f50 2023-07-02T09:00:58.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.415+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f849c0725b0 con 0x7f849c132f50 2023-07-02T09:00:58.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.415+0000 7f8493fff700 1 -- 172.21.15.103:0/4073972430 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f84840098a0 con 0x7f849c132f50 2023-07-02T09:00:58.415 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:58.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f847c0377c0 msgr2=0x7f847c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 --2- 172.21.15.103:0/4073972430 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f847c0377c0 0x7f847c039cd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c132f50 msgr2=0x7f849c1ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 --2- 172.21.15.103:0/4073972430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c132f50 0x7f849c1ba9b0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 shutdown_connections 2023-07-02T09:00:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 --2- 172.21.15.103:0/4073972430 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f847c0377c0 0x7f847c039cd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 --2- 172.21.15.103:0/4073972430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f849c132f50 0x7f849c1ba9b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:58.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 >> 172.21.15.103:0/4073972430 conn(0x7f849c0a2080 msgr2=0x7f849c0a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:58.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 shutdown_connections 2023-07-02T09:00:58.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:58.417+0000 7f84a1c1e700 1 -- 172.21.15.103:0/4073972430 wait complete. 2023-07-02T09:00:58.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:00:59.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:00:59.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.568+0000 7f34d8e9f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:59.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.573+0000 7f34d8e9f700 1 Processor -- start 2023-07-02T09:00:59.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.573+0000 7f34d8e9f700 1 -- start start 2023-07-02T09:00:59.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.574+0000 7f34d8e9f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f34d4132030 0x7f34d4134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:59.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.574+0000 7f34d8e9f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d406df60 con 0x7f34d412f250 2023-07-02T09:00:59.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.574+0000 7f34d8e9f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d409a710 con 0x7f34d4132030 2023-07-02T09:00:59.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.574+0000 7f34d1d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f34d4132030 0x7f34d4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:59.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.574+0000 7f34d259c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f34d412f250 0x7f34d412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57054/0 (socket says 172.21.15.103:57054) 2023-07-02T09:00:59.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.574+0000 7f34d259c700 1 -- 172.21.15.103:0/2618454401 learned_addr learned my addr 172.21.15.103:0/2618454401 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:00:59.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.575+0000 7f34d1d9b700 1 -- 172.21.15.103:0/2618454401 >> v1:172.21.15.103:6789/0 conn(0x7f34d412f250 legacy=0x7f34d412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:59.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.575+0000 7f34d1d9b700 1 -- 172.21.15.103:0/2618454401 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d412ed20 con 0x7f34d4132030 2023-07-02T09:00:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.575+0000 7f34d1d9b700 1 --2- 172.21.15.103:0/2618454401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4132030 0x7f34d4134450 crc :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5925c2f417d699c server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.575+0000 7f34d159a700 1 -- 172.21.15.103:0/2618454401 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f34c4009cd0 con 0x7f34d4132030 2023-07-02T09:00:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.575+0000 7f34d159a700 1 -- 172.21.15.103:0/2618454401 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f34c4009e30 con 0x7f34d4132030 2023-07-02T09:00:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.575+0000 7f34d159a700 1 -- 172.21.15.103:0/2618454401 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f34c400f470 con 0x7f34d4132030 2023-07-02T09:00:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.575+0000 7f34d8e9f700 1 -- 172.21.15.103:0/2618454401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4132030 msgr2=0x7f34d4134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:59.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.575+0000 7f34d8e9f700 1 --2- 172.21.15.103:0/2618454401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4132030 0x7f34d4134450 crc :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:59.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.576+0000 7f34d8e9f700 1 -- 172.21.15.103:0/2618454401 shutdown_connections 2023-07-02T09:00:59.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.576+0000 7f34d8e9f700 1 --2- 172.21.15.103:0/2618454401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4132030 0x7f34d4134450 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:59.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.576+0000 7f34d8e9f700 1 -- 172.21.15.103:0/2618454401 >> 172.21.15.103:0/2618454401 conn(0x7f34d409d810 msgr2=0x7f34d409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:59.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.576+0000 7f34d8e9f700 1 -- 172.21.15.103:0/2618454401 shutdown_connections 2023-07-02T09:00:59.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.576+0000 7f34d8e9f700 1 -- 172.21.15.103:0/2618454401 wait complete. 2023-07-02T09:00:59.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.576+0000 7f34d8e9f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:00:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.577+0000 7f34d8e9f700 1 Processor -- start 2023-07-02T09:00:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.577+0000 7f34d8e9f700 1 -- start start 2023-07-02T09:00:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d8e9f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d412f250 0x7f34d4117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:59.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d8e9f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4128ef0 con 0x7f34d412f250 2023-07-02T09:00:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d412f250 0x7f34d4117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d412f250 0x7f34d4117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54728/0 (socket says 172.21.15.103:54728) 2023-07-02T09:00:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d259c700 1 -- 172.21.15.103:0/67367604 learned_addr learned my addr 172.21.15.103:0/67367604 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:00:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d259c700 1 -- 172.21.15.103:0/67367604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d411b6c0 con 0x7f34d412f250 2023-07-02T09:00:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d259c700 1 --2- 172.21.15.103:0/67367604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d412f250 0x7f34d4117360 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34c2ffd700 1 -- 172.21.15.103:0/67367604 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f34c8005a60 con 0x7f34d412f250 2023-07-02T09:00:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34c2ffd700 1 -- 172.21.15.103:0/67367604 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f34c8005bc0 con 0x7f34d412f250 2023-07-02T09:00:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4118f30 con 0x7f34d412f250 2023-07-02T09:00:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34c2ffd700 1 -- 172.21.15.103:0/67367604 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f34c8005e60 con 0x7f34d412f250 2023-07-02T09:00:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.578+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d4119810 con 0x7f34d412f250 2023-07-02T09:00:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.579+0000 7f34c2ffd700 1 -- 172.21.15.103:0/67367604 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f34c80070b0 con 0x7f34d412f250 2023-07-02T09:00:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.579+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d4128cf0 con 0x7f34d412f250 2023-07-02T09:00:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.579+0000 7f34c2ffd700 1 --2- 172.21.15.103:0/67367604 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f34b40377c0 0x7f34b4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:00:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.579+0000 7f34c2ffd700 1 -- 172.21.15.103:0/67367604 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f34c8046ad0 con 0x7f34d412f250 2023-07-02T09:00:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.580+0000 7f34d1d9b700 1 --2- 172.21.15.103:0/67367604 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f34b40377c0 0x7f34b4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:00:59.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.580+0000 7f34d1d9b700 1 --2- 172.21.15.103:0/67367604 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f34b40377c0 0x7f34b4039cd0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:00:59.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.584+0000 7f34c2ffd700 1 -- 172.21.15.103:0/67367604 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f34c8048e60 con 0x7f34d412f250 2023-07-02T09:00:59.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.775+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f34d4061ea0 con 0x7f34d412f250 2023-07-02T09:00:59.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.775+0000 7f34c2ffd700 1 -- 172.21.15.103:0/67367604 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f34c800a590 con 0x7f34d412f250 2023-07-02T09:00:59.776 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:00:59.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.777+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f34b40377c0 msgr2=0x7f34b4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:59.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.777+0000 7f34d8e9f700 1 --2- 172.21.15.103:0/67367604 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f34b40377c0 0x7f34b4039cd0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:59.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.777+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d412f250 msgr2=0x7f34d4117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:00:59.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.777+0000 7f34d8e9f700 1 --2- 172.21.15.103:0/67367604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d412f250 0x7f34d4117360 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:59.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.777+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 shutdown_connections 2023-07-02T09:00:59.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.777+0000 7f34d8e9f700 1 --2- 172.21.15.103:0/67367604 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f34b40377c0 0x7f34b4039cd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:59.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.777+0000 7f34d8e9f700 1 --2- 172.21.15.103:0/67367604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d412f250 0x7f34d4117360 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:00:59.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.777+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 >> 172.21.15.103:0/67367604 conn(0x7f34d409d810 msgr2=0x7f34d412e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:00:59.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.778+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 shutdown_connections 2023-07-02T09:00:59.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:00:59.778+0000 7f34d8e9f700 1 -- 172.21.15.103:0/67367604 wait complete. 2023-07-02T09:00:59.790 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:00.792 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:00.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.931+0000 7fa3df027700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:00.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.936+0000 7fa3df027700 1 Processor -- start 2023-07-02T09:01:00.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.936+0000 7fa3df027700 1 -- start start 2023-07-02T09:01:00.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3df027700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa3d8098b70 0x7fa3d8138250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:00.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3df027700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d8099b70 con 0x7fa3d80732c0 2023-07-02T09:01:00.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3df027700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d809af80 con 0x7fa3d8098b70 2023-07-02T09:01:00.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3d7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa3d8098b70 0x7fa3d8138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:00.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3dcdc3700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa3d80732c0 0x7fa3d8098460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57056/0 (socket says 172.21.15.103:57056) 2023-07-02T09:01:00.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3dcdc3700 1 -- 172.21.15.103:0/3579088144 learned_addr learned my addr 172.21.15.103:0/3579088144 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:00.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3d7fff700 1 -- 172.21.15.103:0/3579088144 >> v1:172.21.15.103:6789/0 conn(0x7fa3d80732c0 legacy=0x7fa3d8098460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:00.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3d7fff700 1 -- 172.21.15.103:0/3579088144 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d806afa0 con 0x7fa3d8098b70 2023-07-02T09:01:00.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.937+0000 7fa3d7fff700 1 --2- 172.21.15.103:0/3579088144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d8098b70 0x7fa3d8138250 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d353a727e4bb32d2 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:00.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3d77fe700 1 -- 172.21.15.103:0/3579088144 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa3c4009cd0 con 0x7fa3d8098b70 2023-07-02T09:01:00.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3d77fe700 1 -- 172.21.15.103:0/3579088144 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa3c4009e30 con 0x7fa3d8098b70 2023-07-02T09:01:00.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3d77fe700 1 -- 172.21.15.103:0/3579088144 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa3c400e360 con 0x7fa3d8098b70 2023-07-02T09:01:00.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3df027700 1 -- 172.21.15.103:0/3579088144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d8098b70 msgr2=0x7fa3d8138250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:00.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3df027700 1 --2- 172.21.15.103:0/3579088144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d8098b70 0x7fa3d8138250 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:00.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3df027700 1 -- 172.21.15.103:0/3579088144 shutdown_connections 2023-07-02T09:01:00.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3df027700 1 --2- 172.21.15.103:0/3579088144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d8098b70 0x7fa3d8138250 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:00.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3df027700 1 -- 172.21.15.103:0/3579088144 >> 172.21.15.103:0/3579088144 conn(0x7fa3d80a1030 msgr2=0x7fa3d80a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:00.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3df027700 1 -- 172.21.15.103:0/3579088144 shutdown_connections 2023-07-02T09:01:00.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.938+0000 7fa3df027700 1 -- 172.21.15.103:0/3579088144 wait complete. 2023-07-02T09:01:00.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.939+0000 7fa3df027700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:00.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.939+0000 7fa3df027700 1 Processor -- start 2023-07-02T09:01:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.940+0000 7fa3df027700 1 -- start start 2023-07-02T09:01:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.940+0000 7fa3df027700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d80732c0 0x7fa3d81c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.940+0000 7fa3df027700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81aaf40 con 0x7fa3d80732c0 2023-07-02T09:01:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.940+0000 7fa3dcdc3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d80732c0 0x7fa3d81c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.940+0000 7fa3dcdc3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d80732c0 0x7fa3d81c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54740/0 (socket says 172.21.15.103:54740) 2023-07-02T09:01:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.940+0000 7fa3dcdc3700 1 -- 172.21.15.103:0/357987461 learned_addr learned my addr 172.21.15.103:0/357987461 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:00.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.941+0000 7fa3dcdc3700 1 -- 172.21.15.103:0/357987461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8095bc0 con 0x7fa3d80732c0 2023-07-02T09:01:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.941+0000 7fa3dcdc3700 1 --2- 172.21.15.103:0/357987461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d80732c0 0x7fa3d81c3e20 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.941+0000 7fa3d57fa700 1 -- 172.21.15.103:0/357987461 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa3cc005a60 con 0x7fa3d80732c0 2023-07-02T09:01:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.941+0000 7fa3d57fa700 1 -- 172.21.15.103:0/357987461 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa3cc005bc0 con 0x7fa3d80732c0 2023-07-02T09:01:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.941+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d8094a80 con 0x7fa3d80732c0 2023-07-02T09:01:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.941+0000 7fa3d57fa700 1 -- 172.21.15.103:0/357987461 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa3cc005e60 con 0x7fa3d80732c0 2023-07-02T09:01:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.941+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d80952a0 con 0x7fa3d80732c0 2023-07-02T09:01:00.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.942+0000 7fa3d57fa700 1 -- 172.21.15.103:0/357987461 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa3cc00c7b0 con 0x7fa3d80732c0 2023-07-02T09:01:00.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.942+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d8061ea0 con 0x7fa3d80732c0 2023-07-02T09:01:00.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.942+0000 7fa3d57fa700 1 --2- 172.21.15.103:0/357987461 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa3bc0377c0 0x7fa3bc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:00.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.942+0000 7fa3d57fa700 1 -- 172.21.15.103:0/357987461 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa3cc046b70 con 0x7fa3d80732c0 2023-07-02T09:01:00.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.942+0000 7fa3d7fff700 1 --2- 172.21.15.103:0/357987461 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa3bc0377c0 0x7fa3bc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:00.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.943+0000 7fa3d7fff700 1 --2- 172.21.15.103:0/357987461 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa3bc0377c0 0x7fa3bc039cd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:00.947+0000 7fa3d57fa700 1 -- 172.21.15.103:0/357987461 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa3cc00a4a0 con 0x7fa3d80732c0 2023-07-02T09:01:01.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.136+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa3d8071700 con 0x7fa3d80732c0 2023-07-02T09:01:01.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.137+0000 7fa3d57fa700 1 -- 172.21.15.103:0/357987461 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa3cc00a4a0 con 0x7fa3d80732c0 2023-07-02T09:01:01.137 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:01.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa3bc0377c0 msgr2=0x7fa3bc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:01.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 --2- 172.21.15.103:0/357987461 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa3bc0377c0 0x7fa3bc039cd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:01.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d80732c0 msgr2=0x7fa3d81c3e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:01.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 --2- 172.21.15.103:0/357987461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d80732c0 0x7fa3d81c3e20 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:01.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 shutdown_connections 2023-07-02T09:01:01.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 --2- 172.21.15.103:0/357987461 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa3bc0377c0 0x7fa3bc039cd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:01.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 --2- 172.21.15.103:0/357987461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3d80732c0 0x7fa3d81c3e20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:01.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 >> 172.21.15.103:0/357987461 conn(0x7fa3d80a1030 msgr2=0x7fa3d80706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:01.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 shutdown_connections 2023-07-02T09:01:01.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:01.139+0000 7fa3df027700 1 -- 172.21.15.103:0/357987461 wait complete. 2023-07-02T09:01:01.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:02.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:02.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.293+0000 7f9a11157700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:02.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.298+0000 7f9a11157700 1 Processor -- start 2023-07-02T09:01:02.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.298+0000 7f9a11157700 1 -- start start 2023-07-02T09:01:02.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a11157700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9a0c132030 0x7f9a0c134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:02.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a11157700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c06df60 con 0x7f9a0c12f250 2023-07-02T09:01:02.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a11157700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a0c09a710 con 0x7f9a0c132030 2023-07-02T09:01:02.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a0a59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9a0c132030 0x7f9a0c134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:02.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a0a59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9a0c132030 0x7f9a0c134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54742/0 (socket says 172.21.15.103:54742) 2023-07-02T09:01:02.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a0a59c700 1 -- 172.21.15.103:0/2749392373 learned_addr learned my addr 172.21.15.103:0/2749392373 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:02.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a0ad9d700 1 --1- 172.21.15.103:0/2749392373 >> v1:172.21.15.103:6789/0 conn(0x7f9a0c12f250 0x7f9a0c12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57058/0 (socket says 172.21.15.103:57058) 2023-07-02T09:01:02.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a0a59c700 1 -- 172.21.15.103:0/2749392373 >> v1:172.21.15.103:6789/0 conn(0x7f9a0c12f250 legacy=0x7f9a0c12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:02.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a0a59c700 1 -- 172.21.15.103:0/2749392373 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c12ed20 con 0x7f9a0c132030 2023-07-02T09:01:02.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a09d9b700 1 -- 172.21.15.103:0/2749392373 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210422527 0 0) 0x7f9a0c06df60 con 0x7f9a0c12f250 2023-07-02T09:01:02.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a0a59c700 1 --2- 172.21.15.103:0/2749392373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c132030 0x7f9a0c134450 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6938d244808f4e server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:02.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a09d9b700 1 -- 172.21.15.103:0/2749392373 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9a0000d580 con 0x7f9a0c132030 2023-07-02T09:01:02.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a09d9b700 1 -- 172.21.15.103:0/2749392373 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9a0000d6e0 con 0x7f9a0c132030 2023-07-02T09:01:02.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.299+0000 7f9a09d9b700 1 -- 172.21.15.103:0/2749392373 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9a0000d980 con 0x7f9a0c132030 2023-07-02T09:01:02.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.300+0000 7f9a11157700 1 -- 172.21.15.103:0/2749392373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c132030 msgr2=0x7f9a0c134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:02.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.300+0000 7f9a11157700 1 --2- 172.21.15.103:0/2749392373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c132030 0x7f9a0c134450 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:02.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.300+0000 7f9a11157700 1 -- 172.21.15.103:0/2749392373 shutdown_connections 2023-07-02T09:01:02.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.300+0000 7f9a11157700 1 --2- 172.21.15.103:0/2749392373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c132030 0x7f9a0c134450 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:02.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.300+0000 7f9a11157700 1 -- 172.21.15.103:0/2749392373 >> 172.21.15.103:0/2749392373 conn(0x7f9a0c09d810 msgr2=0x7f9a0c09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:02.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.300+0000 7f9a11157700 1 -- 172.21.15.103:0/2749392373 shutdown_connections 2023-07-02T09:01:02.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.300+0000 7f9a11157700 1 -- 172.21.15.103:0/2749392373 wait complete. 2023-07-02T09:01:02.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.300+0000 7f9a11157700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:02.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.302+0000 7f9a11157700 1 Processor -- start 2023-07-02T09:01:02.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.302+0000 7f9a11157700 1 -- start start 2023-07-02T09:01:02.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.302+0000 7f9a11157700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c12f250 0x7f9a0c06fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:02.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.302+0000 7f9a11157700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c071390 con 0x7f9a0c12f250 2023-07-02T09:01:02.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.302+0000 7f9a0ad9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c12f250 0x7f9a0c06fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:02.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.302+0000 7f9a0ad9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c12f250 0x7f9a0c06fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54750/0 (socket says 172.21.15.103:54750) 2023-07-02T09:01:02.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.302+0000 7f9a0ad9d700 1 -- 172.21.15.103:0/564267359 learned_addr learned my addr 172.21.15.103:0/564267359 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:02.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.303+0000 7f9a0ad9d700 1 -- 172.21.15.103:0/564267359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c1292e0 con 0x7f9a0c12f250 2023-07-02T09:01:02.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.303+0000 7f9a0ad9d700 1 --2- 172.21.15.103:0/564267359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c12f250 0x7f9a0c06fe70 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:02.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.303+0000 7f99f37fe700 1 -- 172.21.15.103:0/564267359 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f99fc004b80 con 0x7f9a0c12f250 2023-07-02T09:01:02.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.303+0000 7f99f37fe700 1 -- 172.21.15.103:0/564267359 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f99fc004ce0 con 0x7f9a0c12f250 2023-07-02T09:01:02.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.303+0000 7f99f37fe700 1 -- 172.21.15.103:0/564267359 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f99fc00c4f0 con 0x7f9a0c12f250 2023-07-02T09:01:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.303+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0c128f40 con 0x7f9a0c12f250 2023-07-02T09:01:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.303+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0c1179c0 con 0x7f9a0c12f250 2023-07-02T09:01:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.304+0000 7f99f37fe700 1 -- 172.21.15.103:0/564267359 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f99fc00c650 con 0x7f9a0c12f250 2023-07-02T09:01:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.304+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0c061ea0 con 0x7f9a0c12f250 2023-07-02T09:01:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.304+0000 7f99f37fe700 1 --2- 172.21.15.103:0/564267359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.304+0000 7f99f37fe700 1 -- 172.21.15.103:0/564267359 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f99fc0467f0 con 0x7f9a0c12f250 2023-07-02T09:01:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.304+0000 7f9a0a59c700 1 --2- 172.21.15.103:0/564267359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:02.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.305+0000 7f9a0a59c700 1 --2- 172.21.15.103:0/564267359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:02.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.309+0000 7f99f37fe700 1 -- 172.21.15.103:0/564267359 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f99fc009ae0 con 0x7f9a0c12f250 2023-07-02T09:01:02.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.502+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9a0c1292c0 con 0x7f9a0c12f250 2023-07-02T09:01:02.503 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.502+0000 7f99f37fe700 1 -- 172.21.15.103:0/564267359 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f99fc0090b0 con 0x7f9a0c12f250 2023-07-02T09:01:02.503 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:02.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 msgr2=0x7f99f8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:02.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 --2- 172.21.15.103:0/564267359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:02.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c12f250 msgr2=0x7f9a0c06fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:02.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 --2- 172.21.15.103:0/564267359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c12f250 0x7f9a0c06fe70 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:02.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 shutdown_connections 2023-07-02T09:01:02.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 --2- 172.21.15.103:0/564267359 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:02.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 --2- 172.21.15.103:0/564267359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a0c12f250 0x7f9a0c06fe70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:02.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 >> 172.21.15.103:0/564267359 conn(0x7f9a0c09d810 msgr2=0x7f9a0c12dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:02.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 shutdown_connections 2023-07-02T09:01:02.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:02.505+0000 7f9a11157700 1 -- 172.21.15.103:0/564267359 wait complete. 2023-07-02T09:01:02.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:02.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~77s 2023-07-02T09:01:03.519 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:03.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.657+0000 7f6597ff1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:03.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.662+0000 7f6597ff1700 1 Processor -- start 2023-07-02T09:01:03.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.663+0000 7f6597ff1700 1 -- start start 2023-07-02T09:01:03.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.663+0000 7f6597ff1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f659006f720 0x7f659006fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:03.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.663+0000 7f6597ff1700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6590067780 con 0x7f6590070070 2023-07-02T09:01:03.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.663+0000 7f6597ff1700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6590067f80 con 0x7f659006f720 2023-07-02T09:01:03.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f6595d8d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f659006f720 0x7f659006fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:03.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f659558c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6590070070 0x7f65901307f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36040/0 (socket says 172.21.15.103:36040) 2023-07-02T09:01:03.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f659558c700 1 -- 172.21.15.103:0/3126580192 learned_addr learned my addr 172.21.15.103:0/3126580192 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:03.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f6595d8d700 1 -- 172.21.15.103:0/3126580192 >> v1:172.21.15.103:6789/0 conn(0x7f6590070070 legacy=0x7f65901307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:03.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f6594d8b700 1 -- 172.21.15.103:0/3126580192 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731978930 0 0) 0x7f6590067780 con 0x7f6590070070 2023-07-02T09:01:03.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f6595d8d700 1 -- 172.21.15.103:0/3126580192 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6590093f30 con 0x7f659006f720 2023-07-02T09:01:03.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f6595d8d700 1 --2- 172.21.15.103:0/3126580192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f659006fb30 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfbdfe9fa66e9c7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:03.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f6594d8b700 1 -- 172.21.15.103:0/3126580192 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6584009cd0 con 0x7f659006f720 2023-07-02T09:01:03.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f6594d8b700 1 -- 172.21.15.103:0/3126580192 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6584009e30 con 0x7f659006f720 2023-07-02T09:01:03.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.664+0000 7f6594d8b700 1 -- 172.21.15.103:0/3126580192 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f658400e360 con 0x7f659006f720 2023-07-02T09:01:03.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.665+0000 7f6597ff1700 1 -- 172.21.15.103:0/3126580192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 msgr2=0x7f659006fb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:03.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.665+0000 7f6597ff1700 1 --2- 172.21.15.103:0/3126580192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f659006fb30 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:03.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.665+0000 7f6597ff1700 1 -- 172.21.15.103:0/3126580192 shutdown_connections 2023-07-02T09:01:03.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.665+0000 7f6597ff1700 1 --2- 172.21.15.103:0/3126580192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f659006fb30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:03.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.665+0000 7f6597ff1700 1 -- 172.21.15.103:0/3126580192 >> 172.21.15.103:0/3126580192 conn(0x7f65900a22a0 msgr2=0x7f659006d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:03.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.665+0000 7f6597ff1700 1 -- 172.21.15.103:0/3126580192 shutdown_connections 2023-07-02T09:01:03.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.665+0000 7f6597ff1700 1 -- 172.21.15.103:0/3126580192 wait complete. 2023-07-02T09:01:03.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.665+0000 7f6597ff1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:03.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.666+0000 7f6597ff1700 1 Processor -- start 2023-07-02T09:01:03.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.666+0000 7f6597ff1700 1 -- start start 2023-07-02T09:01:03.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6597ff1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f65901c1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:03.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6597ff1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659009af90 con 0x7f659006f720 2023-07-02T09:01:03.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6595d8d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f65901c1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:03.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6595d8d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f65901c1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46902/0 (socket says 172.21.15.103:46902) 2023-07-02T09:01:03.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6595d8d700 1 -- 172.21.15.103:0/2314886193 learned_addr learned my addr 172.21.15.103:0/2314886193 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:03.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6595d8d700 1 -- 172.21.15.103:0/2314886193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659009bb10 con 0x7f659006f720 2023-07-02T09:01:03.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6595d8d700 1 --2- 172.21.15.103:0/2314886193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f65901c1b20 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:03.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f65827fc700 1 -- 172.21.15.103:0/2314886193 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6584014540 con 0x7f659006f720 2023-07-02T09:01:03.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f65827fc700 1 -- 172.21.15.103:0/2314886193 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f65840146a0 con 0x7f659006f720 2023-07-02T09:01:03.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659009c9f0 con 0x7f659006f720 2023-07-02T09:01:03.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f65827fc700 1 -- 172.21.15.103:0/2314886193 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6584014940 con 0x7f659006f720 2023-07-02T09:01:03.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.667+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659009b450 con 0x7f659006f720 2023-07-02T09:01:03.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.668+0000 7f65827fc700 1 -- 172.21.15.103:0/2314886193 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6584014af0 con 0x7f659006f720 2023-07-02T09:01:03.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.668+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6590061ea0 con 0x7f659006f720 2023-07-02T09:01:03.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.668+0000 7f65827fc700 1 --2- 172.21.15.103:0/2314886193 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65740377b0 0x7f6574039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:03.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.668+0000 7f65827fc700 1 -- 172.21.15.103:0/2314886193 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f65840483b0 con 0x7f659006f720 2023-07-02T09:01:03.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.669+0000 7f659558c700 1 --2- 172.21.15.103:0/2314886193 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65740377b0 0x7f6574039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:03.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.669+0000 7f659558c700 1 --2- 172.21.15.103:0/2314886193 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65740377b0 0x7f6574039cc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:03.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.673+0000 7f65827fc700 1 -- 172.21.15.103:0/2314886193 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f658400fe60 con 0x7f659006f720 2023-07-02T09:01:03.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.864+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6590073ca0 con 0x7f659006f720 2023-07-02T09:01:03.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.864+0000 7f65827fc700 1 -- 172.21.15.103:0/2314886193 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6584010e60 con 0x7f659006f720 2023-07-02T09:01:03.864 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:03.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65740377b0 msgr2=0x7f6574039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:03.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 --2- 172.21.15.103:0/2314886193 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65740377b0 0x7f6574039cc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:03.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 msgr2=0x7f65901c1b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:03.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 --2- 172.21.15.103:0/2314886193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f65901c1b20 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:03.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 shutdown_connections 2023-07-02T09:01:03.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 --2- 172.21.15.103:0/2314886193 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65740377b0 0x7f6574039cc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:03.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 --2- 172.21.15.103:0/2314886193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659006f720 0x7f65901c1b20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:03.868 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 >> 172.21.15.103:0/2314886193 conn(0x7f65900a22a0 msgr2=0x7f659008f670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:03.868 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 shutdown_connections 2023-07-02T09:01:03.868 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:03.866+0000 7f6597ff1700 1 -- 172.21.15.103:0/2314886193 wait complete. 2023-07-02T09:01:03.879 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:04.881 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:05.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.010+0000 7f5b0c57f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:05.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.015+0000 7f5b0c57f700 1 Processor -- start 2023-07-02T09:01:05.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.015+0000 7f5b0c57f700 1 -- start start 2023-07-02T09:01:05.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0c57f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5b04132f50 0x7f5b04133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:05.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0c57f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0409e260 con 0x7f5b0409c580 2023-07-02T09:01:05.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0c57f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b040a1640 con 0x7f5b04132f50 2023-07-02T09:01:05.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0a31b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5b04132f50 0x7f5b04133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:05.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0a31b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5b04132f50 0x7f5b04133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46904/0 (socket says 172.21.15.103:46904) 2023-07-02T09:01:05.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b09b1a700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f5b0409c580 0x7f5b0409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36054/0 (socket says 172.21.15.103:36054) 2023-07-02T09:01:05.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0a31b700 1 -- 172.21.15.103:0/1996770029 learned_addr learned my addr 172.21.15.103:0/1996770029 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:05.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0a31b700 1 -- 172.21.15.103:0/1996770029 >> v1:172.21.15.103:6789/0 conn(0x7f5b0409c580 legacy=0x7f5b0409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:05.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0a31b700 1 -- 172.21.15.103:0/1996770029 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b04132a20 con 0x7f5b04132f50 2023-07-02T09:01:05.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b09319700 1 -- 172.21.15.103:0/1996770029 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341339030 0 0) 0x7f5b0409e260 con 0x7f5b0409c580 2023-07-02T09:01:05.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b0a31b700 1 --2- 172.21.15.103:0/1996770029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b04132f50 0x7f5b04133360 crc :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a34f9086efb102 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:05.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b09319700 1 -- 172.21.15.103:0/1996770029 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5af4009d90 con 0x7f5b04132f50 2023-07-02T09:01:05.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b09319700 1 -- 172.21.15.103:0/1996770029 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5af400e230 con 0x7f5b04132f50 2023-07-02T09:01:05.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.016+0000 7f5b09319700 1 -- 172.21.15.103:0/1996770029 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5af400e4d0 con 0x7f5b04132f50 2023-07-02T09:01:05.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.017+0000 7f5b0c57f700 1 -- 172.21.15.103:0/1996770029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b04132f50 msgr2=0x7f5b04133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:05.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.017+0000 7f5b0c57f700 1 --2- 172.21.15.103:0/1996770029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b04132f50 0x7f5b04133360 crc :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:05.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.017+0000 7f5b0c57f700 1 -- 172.21.15.103:0/1996770029 shutdown_connections 2023-07-02T09:01:05.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.017+0000 7f5b0c57f700 1 --2- 172.21.15.103:0/1996770029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b04132f50 0x7f5b04133360 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:05.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.017+0000 7f5b0c57f700 1 -- 172.21.15.103:0/1996770029 >> 172.21.15.103:0/1996770029 conn(0x7f5b040a2080 msgr2=0x7f5b040a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:05.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.017+0000 7f5b0c57f700 1 -- 172.21.15.103:0/1996770029 shutdown_connections 2023-07-02T09:01:05.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.017+0000 7f5b0c57f700 1 -- 172.21.15.103:0/1996770029 wait complete. 2023-07-02T09:01:05.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.017+0000 7f5b0c57f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:05.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.018+0000 7f5b0c57f700 1 Processor -- start 2023-07-02T09:01:05.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.018+0000 7f5b0c57f700 1 -- start start 2023-07-02T09:01:05.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5b0c57f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0409c580 0x7f5b041baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:05.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5b0c57f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b041ba610 con 0x7f5b0409c580 2023-07-02T09:01:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5b0a31b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0409c580 0x7f5b041baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5b0a31b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0409c580 0x7f5b041baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46914/0 (socket says 172.21.15.103:46914) 2023-07-02T09:01:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5b0a31b700 1 -- 172.21.15.103:0/3293072403 learned_addr learned my addr 172.21.15.103:0/3293072403 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5b0a31b700 1 -- 172.21.15.103:0/3293072403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0406dfc0 con 0x7f5b0409c580 2023-07-02T09:01:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5b0a31b700 1 --2- 172.21.15.103:0/3293072403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0409c580 0x7f5b041baa60 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5afaffd700 1 -- 172.21.15.103:0/3293072403 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5af4014410 con 0x7f5b0409c580 2023-07-02T09:01:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.019+0000 7f5afaffd700 1 -- 172.21.15.103:0/3293072403 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5af4014570 con 0x7f5b0409c580 2023-07-02T09:01:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.020+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b041ba270 con 0x7f5b0409c580 2023-07-02T09:01:05.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.020+0000 7f5afaffd700 1 -- 172.21.15.103:0/3293072403 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5af4014810 con 0x7f5b0409c580 2023-07-02T09:01:05.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.020+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0406dba0 con 0x7f5b0409c580 2023-07-02T09:01:05.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.020+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b041ba460 con 0x7f5b0409c580 2023-07-02T09:01:05.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.020+0000 7f5afaffd700 1 -- 172.21.15.103:0/3293072403 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f5af40149c0 con 0x7f5b0409c580 2023-07-02T09:01:05.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.021+0000 7f5afaffd700 1 --2- 172.21.15.103:0/3293072403 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ae80377c0 0x7f5ae8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:05.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.021+0000 7f5afaffd700 1 -- 172.21.15.103:0/3293072403 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f5af40482f0 con 0x7f5b0409c580 2023-07-02T09:01:05.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.021+0000 7f5b09b1a700 1 --2- 172.21.15.103:0/3293072403 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ae80377c0 0x7f5ae8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:05.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.021+0000 7f5b09b1a700 1 --2- 172.21.15.103:0/3293072403 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ae80377c0 0x7f5ae8039cd0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:05.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.025+0000 7f5afaffd700 1 -- 172.21.15.103:0/3293072403 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f5af400f8f0 con 0x7f5b0409c580 2023-07-02T09:01:05.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.215+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5b04061ea0 con 0x7f5b0409c580 2023-07-02T09:01:05.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.216+0000 7f5afaffd700 1 -- 172.21.15.103:0/3293072403 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f5af400f710 con 0x7f5b0409c580 2023-07-02T09:01:05.216 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:05.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ae80377c0 msgr2=0x7f5ae8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:05.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 --2- 172.21.15.103:0/3293072403 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ae80377c0 0x7f5ae8039cd0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:05.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0409c580 msgr2=0x7f5b041baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 --2- 172.21.15.103:0/3293072403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0409c580 0x7f5b041baa60 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 shutdown_connections 2023-07-02T09:01:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 --2- 172.21.15.103:0/3293072403 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ae80377c0 0x7f5ae8039cd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 --2- 172.21.15.103:0/3293072403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b0409c580 0x7f5b041baa60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 >> 172.21.15.103:0/3293072403 conn(0x7f5b040a2080 msgr2=0x7f5b04130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 shutdown_connections 2023-07-02T09:01:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:05.218+0000 7f5b0c57f700 1 -- 172.21.15.103:0/3293072403 wait complete. 2023-07-02T09:01:05.230 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:06.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:06.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.370+0000 7f0df45e4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:06.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.375+0000 7f0df45e4700 1 Processor -- start 2023-07-02T09:01:06.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.375+0000 7f0df45e4700 1 -- start start 2023-07-02T09:01:06.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.376+0000 7f0df45e4700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0dec12f250 0x7f0dec12f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:06.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.376+0000 7f0df45e4700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dec06df60 con 0x7f0dec098a00 2023-07-02T09:01:06.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.376+0000 7f0df45e4700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dec09a710 con 0x7f0dec12f250 2023-07-02T09:01:06.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.376+0000 7f0df2380700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0dec12f250 0x7f0dec12f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:06.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.376+0000 7f0df2380700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0dec12f250 0x7f0dec12f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46928/0 (socket says 172.21.15.103:46928) 2023-07-02T09:01:06.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.376+0000 7f0df1b7f700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f0dec098a00 0x7f0dec098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36068/0 (socket says 172.21.15.103:36068) 2023-07-02T09:01:06.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.376+0000 7f0df2380700 1 -- 172.21.15.103:0/2600776989 learned_addr learned my addr 172.21.15.103:0/2600776989 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:06.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.377+0000 7f0df2380700 1 -- 172.21.15.103:0/2600776989 >> v1:172.21.15.103:6789/0 conn(0x7f0dec098a00 legacy=0x7f0dec098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:06.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.377+0000 7f0df2380700 1 -- 172.21.15.103:0/2600776989 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec12ed20 con 0x7f0dec12f250 2023-07-02T09:01:06.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.377+0000 7f0df2380700 1 --2- 172.21.15.103:0/2600776989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec12f250 0x7f0dec12f660 crc :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e2d8d26886bc73 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:06.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.377+0000 7f0df137e700 1 -- 172.21.15.103:0/2600776989 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0ddc009d90 con 0x7f0dec12f250 2023-07-02T09:01:06.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.377+0000 7f0df137e700 1 -- 172.21.15.103:0/2600776989 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0ddc00e230 con 0x7f0dec12f250 2023-07-02T09:01:06.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.377+0000 7f0df137e700 1 -- 172.21.15.103:0/2600776989 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0ddc00e4d0 con 0x7f0dec12f250 2023-07-02T09:01:06.380 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.377+0000 7f0df45e4700 1 -- 172.21.15.103:0/2600776989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec12f250 msgr2=0x7f0dec12f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:06.380 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.377+0000 7f0df45e4700 1 --2- 172.21.15.103:0/2600776989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec12f250 0x7f0dec12f660 crc :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:06.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.378+0000 7f0df45e4700 1 -- 172.21.15.103:0/2600776989 shutdown_connections 2023-07-02T09:01:06.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.378+0000 7f0df45e4700 1 --2- 172.21.15.103:0/2600776989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec12f250 0x7f0dec12f660 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:06.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.378+0000 7f0df45e4700 1 -- 172.21.15.103:0/2600776989 >> 172.21.15.103:0/2600776989 conn(0x7f0dec09d810 msgr2=0x7f0dec09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:06.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.378+0000 7f0df45e4700 1 -- 172.21.15.103:0/2600776989 shutdown_connections 2023-07-02T09:01:06.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.378+0000 7f0df45e4700 1 -- 172.21.15.103:0/2600776989 wait complete. 2023-07-02T09:01:06.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.378+0000 7f0df45e4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:06.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.379+0000 7f0df45e4700 1 Processor -- start 2023-07-02T09:01:06.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.379+0000 7f0df45e4700 1 -- start start 2023-07-02T09:01:06.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.379+0000 7f0df45e4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec098a00 0x7f0dec11a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:06.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.379+0000 7f0df45e4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec11da80 con 0x7f0dec098a00 2023-07-02T09:01:06.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.379+0000 7f0df2380700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec098a00 0x7f0dec11a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:06.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.379+0000 7f0df2380700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec098a00 0x7f0dec11a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46940/0 (socket says 172.21.15.103:46940) 2023-07-02T09:01:06.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.379+0000 7f0df2380700 1 -- 172.21.15.103:0/2522692360 learned_addr learned my addr 172.21.15.103:0/2522692360 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:06.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.380+0000 7f0df2380700 1 -- 172.21.15.103:0/2522692360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec12d110 con 0x7f0dec098a00 2023-07-02T09:01:06.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.380+0000 7f0df2380700 1 --2- 172.21.15.103:0/2522692360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec098a00 0x7f0dec11a6d0 crc :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:06.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.380+0000 7f0de2ffd700 1 -- 172.21.15.103:0/2522692360 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0ddc014410 con 0x7f0dec098a00 2023-07-02T09:01:06.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.380+0000 7f0de2ffd700 1 -- 172.21.15.103:0/2522692360 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0ddc014570 con 0x7f0dec098a00 2023-07-02T09:01:06.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.380+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dec12d8a0 con 0x7f0dec098a00 2023-07-02T09:01:06.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.380+0000 7f0de2ffd700 1 -- 172.21.15.103:0/2522692360 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0ddc014810 con 0x7f0dec098a00 2023-07-02T09:01:06.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.380+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dec11c9a0 con 0x7f0dec098a00 2023-07-02T09:01:06.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.381+0000 7f0de2ffd700 1 -- 172.21.15.103:0/2522692360 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f0ddc0149c0 con 0x7f0dec098a00 2023-07-02T09:01:06.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.381+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dec09c800 con 0x7f0dec098a00 2023-07-02T09:01:06.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.381+0000 7f0de2ffd700 1 --2- 172.21.15.103:0/2522692360 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0dd00377c0 0x7f0dd0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:06.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.381+0000 7f0de2ffd700 1 -- 172.21.15.103:0/2522692360 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f0ddc048280 con 0x7f0dec098a00 2023-07-02T09:01:06.385 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.381+0000 7f0df1b7f700 1 --2- 172.21.15.103:0/2522692360 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0dd00377c0 0x7f0dd0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:06.385 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.382+0000 7f0df1b7f700 1 --2- 172.21.15.103:0/2522692360 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0dd00377c0 0x7f0dd0039cd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:06.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.386+0000 7f0de2ffd700 1 -- 172.21.15.103:0/2522692360 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f0ddc00f8f0 con 0x7f0dec098a00 2023-07-02T09:01:06.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.573+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0dec11bbd0 con 0x7f0dec098a00 2023-07-02T09:01:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.573+0000 7f0de2ffd700 1 -- 172.21.15.103:0/2522692360 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f0ddc00f710 con 0x7f0dec098a00 2023-07-02T09:01:06.574 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:06.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0dd00377c0 msgr2=0x7f0dd0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:06.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 --2- 172.21.15.103:0/2522692360 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0dd00377c0 0x7f0dd0039cd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:06.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec098a00 msgr2=0x7f0dec11a6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:06.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 --2- 172.21.15.103:0/2522692360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec098a00 0x7f0dec11a6d0 crc :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:06.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 shutdown_connections 2023-07-02T09:01:06.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 --2- 172.21.15.103:0/2522692360 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0dd00377c0 0x7f0dd0039cd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:06.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 --2- 172.21.15.103:0/2522692360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dec098a00 0x7f0dec11a6d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:06.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 >> 172.21.15.103:0/2522692360 conn(0x7f0dec09d810 msgr2=0x7f0dec09eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:06.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 shutdown_connections 2023-07-02T09:01:06.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:06.575+0000 7f0df45e4700 1 -- 172.21.15.103:0/2522692360 wait complete. 2023-07-02T09:01:06.588 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:07.589 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:07.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.728+0000 7f59b63fe700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:07.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.733+0000 7f59b63fe700 1 Processor -- start 2023-07-02T09:01:07.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.733+0000 7f59b63fe700 1 -- start start 2023-07-02T09:01:07.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.733+0000 7f59b63fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f59b0132030 0x7f59b0134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:07.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.733+0000 7f59b63fe700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b006df60 con 0x7f59b012f250 2023-07-02T09:01:07.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.733+0000 7f59b63fe700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b009a710 con 0x7f59b0132030 2023-07-02T09:01:07.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59affff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f59b012f250 0x7f59b012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36070/0 (socket says 172.21.15.103:36070) 2023-07-02T09:01:07.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59affff700 1 -- 172.21.15.103:0/800619732 learned_addr learned my addr 172.21.15.103:0/800619732 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:07.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59af7fe700 1 --2- 172.21.15.103:0/800619732 >> v2:172.21.15.103:3300/0 conn(0x7f59b0132030 0x7f59b0134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:07.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59af7fe700 1 -- 172.21.15.103:0/800619732 >> v1:172.21.15.103:6789/0 conn(0x7f59b012f250 legacy=0x7f59b012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:07.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59af7fe700 1 -- 172.21.15.103:0/800619732 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b012ed20 con 0x7f59b0132030 2023-07-02T09:01:07.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59af7fe700 1 --2- 172.21.15.103:0/800619732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b0132030 0x7f59b0134450 crc :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28c7fa701f21f9c server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:07.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59aeffd700 1 -- 172.21.15.103:0/800619732 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f59a0009cd0 con 0x7f59b0132030 2023-07-02T09:01:07.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59aeffd700 1 -- 172.21.15.103:0/800619732 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f59a0009e30 con 0x7f59b0132030 2023-07-02T09:01:07.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.734+0000 7f59aeffd700 1 -- 172.21.15.103:0/800619732 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f59a000e360 con 0x7f59b0132030 2023-07-02T09:01:07.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.735+0000 7f59b63fe700 1 -- 172.21.15.103:0/800619732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b0132030 msgr2=0x7f59b0134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:07.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.735+0000 7f59b63fe700 1 --2- 172.21.15.103:0/800619732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b0132030 0x7f59b0134450 crc :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:07.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.735+0000 7f59b63fe700 1 -- 172.21.15.103:0/800619732 shutdown_connections 2023-07-02T09:01:07.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.735+0000 7f59b63fe700 1 --2- 172.21.15.103:0/800619732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b0132030 0x7f59b0134450 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:07.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.735+0000 7f59b63fe700 1 -- 172.21.15.103:0/800619732 >> 172.21.15.103:0/800619732 conn(0x7f59b009d810 msgr2=0x7f59b009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:07.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.735+0000 7f59b63fe700 1 -- 172.21.15.103:0/800619732 shutdown_connections 2023-07-02T09:01:07.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.735+0000 7f59b63fe700 1 -- 172.21.15.103:0/800619732 wait complete. 2023-07-02T09:01:07.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.735+0000 7f59b63fe700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:07.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.736+0000 7f59b63fe700 1 Processor -- start 2023-07-02T09:01:07.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.737+0000 7f59b63fe700 1 -- start start 2023-07-02T09:01:07.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.737+0000 7f59b63fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b012f250 0x7f59b0070440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:07.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.737+0000 7f59b63fe700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b0071940 con 0x7f59b012f250 2023-07-02T09:01:07.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.737+0000 7f59affff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b012f250 0x7f59b0070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:07.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.738+0000 7f59affff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b012f250 0x7f59b0070440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46962/0 (socket says 172.21.15.103:46962) 2023-07-02T09:01:07.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.738+0000 7f59affff700 1 -- 172.21.15.103:0/3339849420 learned_addr learned my addr 172.21.15.103:0/3339849420 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:07.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.738+0000 7f59affff700 1 -- 172.21.15.103:0/3339849420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b00716a0 con 0x7f59b012f250 2023-07-02T09:01:07.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.738+0000 7f59affff700 1 --2- 172.21.15.103:0/3339849420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b012f250 0x7f59b0070440 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:07.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.740+0000 7f59acff9700 1 -- 172.21.15.103:0/3339849420 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f59a4005a60 con 0x7f59b012f250 2023-07-02T09:01:07.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.740+0000 7f59acff9700 1 -- 172.21.15.103:0/3339849420 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f59a4005bc0 con 0x7f59b012f250 2023-07-02T09:01:07.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.740+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b006fe80 con 0x7f59b012f250 2023-07-02T09:01:07.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.740+0000 7f59acff9700 1 -- 172.21.15.103:0/3339849420 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f59a4005e60 con 0x7f59b012f250 2023-07-02T09:01:07.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.740+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b01364c0 con 0x7f59b012f250 2023-07-02T09:01:07.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.741+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b0061ea0 con 0x7f59b012f250 2023-07-02T09:01:07.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.744+0000 7f59acff9700 1 -- 172.21.15.103:0/3339849420 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f59a40070b0 con 0x7f59b012f250 2023-07-02T09:01:07.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.744+0000 7f59acff9700 1 --2- 172.21.15.103:0/3339849420 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f599c0377c0 0x7f599c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:07.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.744+0000 7f59acff9700 1 -- 172.21.15.103:0/3339849420 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f59a4046d60 con 0x7f59b012f250 2023-07-02T09:01:07.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.744+0000 7f59af7fe700 1 --2- 172.21.15.103:0/3339849420 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f599c0377c0 0x7f599c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:07.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.745+0000 7f59af7fe700 1 --2- 172.21.15.103:0/3339849420 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f599c0377c0 0x7f599c039cd0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:07.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.748+0000 7f59acff9700 1 -- 172.21.15.103:0/3339849420 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f59a400d510 con 0x7f59b012f250 2023-07-02T09:01:07.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.945+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f59b0073ca0 con 0x7f59b012f250 2023-07-02T09:01:07.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.945+0000 7f59acff9700 1 -- 172.21.15.103:0/3339849420 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f59a400d150 con 0x7f59b012f250 2023-07-02T09:01:07.945 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:07.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f599c0377c0 msgr2=0x7f599c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:07.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 --2- 172.21.15.103:0/3339849420 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f599c0377c0 0x7f599c039cd0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:07.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b012f250 msgr2=0x7f59b0070440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:07.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 --2- 172.21.15.103:0/3339849420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b012f250 0x7f59b0070440 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:07.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 shutdown_connections 2023-07-02T09:01:07.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 --2- 172.21.15.103:0/3339849420 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f599c0377c0 0x7f599c039cd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:07.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 --2- 172.21.15.103:0/3339849420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b012f250 0x7f59b0070440 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:07.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 >> 172.21.15.103:0/3339849420 conn(0x7f59b009d810 msgr2=0x7f59b012e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:07.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 shutdown_connections 2023-07-02T09:01:07.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:07.947+0000 7f59b63fe700 1 -- 172.21.15.103:0/3339849420 wait complete. 2023-07-02T09:01:07.960 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~83s 2023-07-02T09:01:08.961 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:09.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.100+0000 7f5adbae1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:09.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.105+0000 7f5adbae1700 1 Processor -- start 2023-07-02T09:01:09.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.105+0000 7f5adbae1700 1 -- start start 2023-07-02T09:01:09.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5adbae1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5ad4132f50 0x7f5ad4133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:09.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5adbae1700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad409e260 con 0x7f5ad409c580 2023-07-02T09:01:09.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5adbae1700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad40a1640 con 0x7f5ad4132f50 2023-07-02T09:01:09.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad987d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5ad4132f50 0x7f5ad4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:09.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad907c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f5ad409c580 0x7f5ad409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36082/0 (socket says 172.21.15.103:36082) 2023-07-02T09:01:09.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad907c700 1 -- 172.21.15.103:0/973747910 learned_addr learned my addr 172.21.15.103:0/973747910 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad987d700 1 -- 172.21.15.103:0/973747910 >> v1:172.21.15.103:6789/0 conn(0x7f5ad409c580 legacy=0x7f5ad409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad987d700 1 -- 172.21.15.103:0/973747910 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4132a20 con 0x7f5ad4132f50 2023-07-02T09:01:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad987d700 1 --2- 172.21.15.103:0/973747910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad4132f50 0x7f5ad4133360 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb31331769612c35 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad887b700 1 -- 172.21.15.103:0/973747910 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5ad0009cd0 con 0x7f5ad4132f50 2023-07-02T09:01:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad887b700 1 -- 172.21.15.103:0/973747910 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5ad0009e30 con 0x7f5ad4132f50 2023-07-02T09:01:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.106+0000 7f5ad887b700 1 -- 172.21.15.103:0/973747910 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5ad000e360 con 0x7f5ad4132f50 2023-07-02T09:01:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.107+0000 7f5adbae1700 1 -- 172.21.15.103:0/973747910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad4132f50 msgr2=0x7f5ad4133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.107+0000 7f5adbae1700 1 --2- 172.21.15.103:0/973747910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad4132f50 0x7f5ad4133360 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.107+0000 7f5adbae1700 1 -- 172.21.15.103:0/973747910 shutdown_connections 2023-07-02T09:01:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.107+0000 7f5adbae1700 1 --2- 172.21.15.103:0/973747910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad4132f50 0x7f5ad4133360 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.107+0000 7f5adbae1700 1 -- 172.21.15.103:0/973747910 >> 172.21.15.103:0/973747910 conn(0x7f5ad40a2080 msgr2=0x7f5ad40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.107+0000 7f5adbae1700 1 -- 172.21.15.103:0/973747910 shutdown_connections 2023-07-02T09:01:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.107+0000 7f5adbae1700 1 -- 172.21.15.103:0/973747910 wait complete. 2023-07-02T09:01:09.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.107+0000 7f5adbae1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:09.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.108+0000 7f5adbae1700 1 Processor -- start 2023-07-02T09:01:09.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.108+0000 7f5adbae1700 1 -- start start 2023-07-02T09:01:09.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.109+0000 7f5adbae1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad409c580 0x7f5ad412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:09.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.109+0000 7f5adbae1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad406f9d0 con 0x7f5ad409c580 2023-07-02T09:01:09.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.109+0000 7f5ad987d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad409c580 0x7f5ad412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:09.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.109+0000 7f5ad987d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad409c580 0x7f5ad412b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46982/0 (socket says 172.21.15.103:46982) 2023-07-02T09:01:09.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.109+0000 7f5ad987d700 1 -- 172.21.15.103:0/3781889586 learned_addr learned my addr 172.21.15.103:0/3781889586 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:09.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.109+0000 7f5ad987d700 1 -- 172.21.15.103:0/3781889586 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4133a00 con 0x7f5ad409c580 2023-07-02T09:01:09.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.109+0000 7f5ad987d700 1 --2- 172.21.15.103:0/3781889586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad409c580 0x7f5ad412b7b0 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:09.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.110+0000 7f5aca7fc700 1 -- 172.21.15.103:0/3781889586 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5ad0013530 con 0x7f5ad409c580 2023-07-02T09:01:09.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.110+0000 7f5aca7fc700 1 -- 172.21.15.103:0/3781889586 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5ad0013690 con 0x7f5ad409c580 2023-07-02T09:01:09.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.110+0000 7f5aca7fc700 1 -- 172.21.15.103:0/3781889586 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5ad0013930 con 0x7f5ad409c580 2023-07-02T09:01:09.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.110+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad406d160 con 0x7f5ad409c580 2023-07-02T09:01:09.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.110+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad406da70 con 0x7f5ad409c580 2023-07-02T09:01:09.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.110+0000 7f5aca7fc700 1 -- 172.21.15.103:0/3781889586 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f5ad0013ae0 con 0x7f5ad409c580 2023-07-02T09:01:09.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.111+0000 7f5aca7fc700 1 --2- 172.21.15.103:0/3781889586 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ab80377c0 0x7f5ab8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:09.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.111+0000 7f5aca7fc700 1 -- 172.21.15.103:0/3781889586 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f5ad0018040 con 0x7f5ad409c580 2023-07-02T09:01:09.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.111+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad4061ea0 con 0x7f5ad409c580 2023-07-02T09:01:09.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.111+0000 7f5ad907c700 1 --2- 172.21.15.103:0/3781889586 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ab80377c0 0x7f5ab8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:09.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.111+0000 7f5ad907c700 1 --2- 172.21.15.103:0/3781889586 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ab80377c0 0x7f5ab8039cd0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:09.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.116+0000 7f5aca7fc700 1 -- 172.21.15.103:0/3781889586 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f5ad0018dd0 con 0x7f5ad409c580 2023-07-02T09:01:09.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.305+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5ad40725b0 con 0x7f5ad409c580 2023-07-02T09:01:09.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.305+0000 7f5aca7fc700 1 -- 172.21.15.103:0/3781889586 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f5ad00189f0 con 0x7f5ad409c580 2023-07-02T09:01:09.306 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:09.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ab80377c0 msgr2=0x7f5ab8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:09.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 --2- 172.21.15.103:0/3781889586 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ab80377c0 0x7f5ab8039cd0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:09.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad409c580 msgr2=0x7f5ad412b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:09.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 --2- 172.21.15.103:0/3781889586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad409c580 0x7f5ad412b7b0 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:09.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 shutdown_connections 2023-07-02T09:01:09.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 --2- 172.21.15.103:0/3781889586 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5ab80377c0 0x7f5ab8039cd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:09.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 --2- 172.21.15.103:0/3781889586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ad409c580 0x7f5ad412b7b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:09.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 >> 172.21.15.103:0/3781889586 conn(0x7f5ad40a2080 msgr2=0x7f5ad41babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:09.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 shutdown_connections 2023-07-02T09:01:09.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:09.307+0000 7f5adbae1700 1 -- 172.21.15.103:0/3781889586 wait complete. 2023-07-02T09:01:09.320 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:10.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:10.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.460+0000 7f20483fa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:10.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.465+0000 7f20483fa700 1 Processor -- start 2023-07-02T09:01:10.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.465+0000 7f20483fa700 1 -- start start 2023-07-02T09:01:10.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f20483fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2040132f50 0x7f2040133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:10.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f20483fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204009e260 con 0x7f204009c580 2023-07-02T09:01:10.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f20483fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20400a1640 con 0x7f2040132f50 2023-07-02T09:01:10.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f2046196700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2040132f50 0x7f2040133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:10.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f2045995700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f204009c580 0x7f204009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36096/0 (socket says 172.21.15.103:36096) 2023-07-02T09:01:10.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f2045995700 1 -- 172.21.15.103:0/2205855738 learned_addr learned my addr 172.21.15.103:0/2205855738 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:10.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f2046196700 1 -- 172.21.15.103:0/2205855738 >> v1:172.21.15.103:6789/0 conn(0x7f204009c580 legacy=0x7f204009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:10.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f2046196700 1 -- 172.21.15.103:0/2205855738 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2040132a20 con 0x7f2040132f50 2023-07-02T09:01:10.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.466+0000 7f2046196700 1 --2- 172.21.15.103:0/2205855738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2040132f50 0x7f2040133360 crc :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4223b60164a0fb6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:10.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f2045194700 1 -- 172.21.15.103:0/2205855738 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2030009cd0 con 0x7f2040132f50 2023-07-02T09:01:10.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f2045194700 1 -- 172.21.15.103:0/2205855738 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2030009e30 con 0x7f2040132f50 2023-07-02T09:01:10.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f2045194700 1 -- 172.21.15.103:0/2205855738 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f203000e360 con 0x7f2040132f50 2023-07-02T09:01:10.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f20483fa700 1 -- 172.21.15.103:0/2205855738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2040132f50 msgr2=0x7f2040133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:10.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f20483fa700 1 --2- 172.21.15.103:0/2205855738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2040132f50 0x7f2040133360 crc :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:10.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f20483fa700 1 -- 172.21.15.103:0/2205855738 shutdown_connections 2023-07-02T09:01:10.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f20483fa700 1 --2- 172.21.15.103:0/2205855738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2040132f50 0x7f2040133360 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:10.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f20483fa700 1 -- 172.21.15.103:0/2205855738 >> 172.21.15.103:0/2205855738 conn(0x7f20400a2080 msgr2=0x7f20400a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:10.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f20483fa700 1 -- 172.21.15.103:0/2205855738 shutdown_connections 2023-07-02T09:01:10.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.467+0000 7f20483fa700 1 -- 172.21.15.103:0/2205855738 wait complete. 2023-07-02T09:01:10.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.468+0000 7f20483fa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:10.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.469+0000 7f20483fa700 1 Processor -- start 2023-07-02T09:01:10.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.469+0000 7f20483fa700 1 -- start start 2023-07-02T09:01:10.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.469+0000 7f20483fa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204009c580 0x7f204012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:10.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.469+0000 7f20483fa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204006f9d0 con 0x7f204009c580 2023-07-02T09:01:10.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.469+0000 7f2046196700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204009c580 0x7f204012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:10.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.469+0000 7f2046196700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204009c580 0x7f204012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47004/0 (socket says 172.21.15.103:47004) 2023-07-02T09:01:10.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.469+0000 7f2046196700 1 -- 172.21.15.103:0/725882864 learned_addr learned my addr 172.21.15.103:0/725882864 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.470+0000 7f2046196700 1 -- 172.21.15.103:0/725882864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2040133a00 con 0x7f204009c580 2023-07-02T09:01:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.470+0000 7f2046196700 1 --2- 172.21.15.103:0/725882864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204009c580 0x7f204012b7b0 crc :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.470+0000 7f2036ffd700 1 -- 172.21.15.103:0/725882864 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2030013530 con 0x7f204009c580 2023-07-02T09:01:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.470+0000 7f2036ffd700 1 -- 172.21.15.103:0/725882864 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2030013690 con 0x7f204009c580 2023-07-02T09:01:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.470+0000 7f2036ffd700 1 -- 172.21.15.103:0/725882864 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2030013930 con 0x7f204009c580 2023-07-02T09:01:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.470+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204006d160 con 0x7f204009c580 2023-07-02T09:01:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.470+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204006da70 con 0x7f204009c580 2023-07-02T09:01:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.471+0000 7f2036ffd700 1 -- 172.21.15.103:0/725882864 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2030013ae0 con 0x7f204009c580 2023-07-02T09:01:10.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.471+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2040061ea0 con 0x7f204009c580 2023-07-02T09:01:10.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.471+0000 7f2036ffd700 1 --2- 172.21.15.103:0/725882864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f20240377c0 0x7f2024039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:10.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.471+0000 7f2036ffd700 1 -- 172.21.15.103:0/725882864 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2030018040 con 0x7f204009c580 2023-07-02T09:01:10.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.471+0000 7f2045995700 1 --2- 172.21.15.103:0/725882864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f20240377c0 0x7f2024039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:10.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.472+0000 7f2045995700 1 --2- 172.21.15.103:0/725882864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f20240377c0 0x7f2024039cd0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:10.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.476+0000 7f2036ffd700 1 -- 172.21.15.103:0/725882864 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2030018dd0 con 0x7f204009c580 2023-07-02T09:01:10.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.667+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f20400725b0 con 0x7f204009c580 2023-07-02T09:01:10.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.667+0000 7f2036ffd700 1 -- 172.21.15.103:0/725882864 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f20300189f0 con 0x7f204009c580 2023-07-02T09:01:10.668 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:10.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f20240377c0 msgr2=0x7f2024039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:10.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 --2- 172.21.15.103:0/725882864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f20240377c0 0x7f2024039cd0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:10.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204009c580 msgr2=0x7f204012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:10.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 --2- 172.21.15.103:0/725882864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204009c580 0x7f204012b7b0 crc :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:10.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 shutdown_connections 2023-07-02T09:01:10.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 --2- 172.21.15.103:0/725882864 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f20240377c0 0x7f2024039cd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:10.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 --2- 172.21.15.103:0/725882864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204009c580 0x7f204012b7b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:10.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 >> 172.21.15.103:0/725882864 conn(0x7f20400a2080 msgr2=0x7f20401babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:10.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 shutdown_connections 2023-07-02T09:01:10.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:10.669+0000 7f20483fa700 1 -- 172.21.15.103:0/725882864 wait complete. 2023-07-02T09:01:10.682 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:11.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:11.823 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.819+0000 7f1e89788700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:11.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.824+0000 7f1e89788700 1 Processor -- start 2023-07-02T09:01:11.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.824+0000 7f1e89788700 1 -- start start 2023-07-02T09:01:11.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e89788700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e8409d8a0 0x7f1e8409dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:11.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e89788700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8409f240 con 0x7f1e8409e1f0 2023-07-02T09:01:11.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e89788700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8409ed50 con 0x7f1e8409d8a0 2023-07-02T09:01:11.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e8409d8a0 0x7f1e8409dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:11.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e827fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1e8409e1f0 0x7f1e84132a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36104/0 (socket says 172.21.15.103:36104) 2023-07-02T09:01:11.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e8409d8a0 0x7f1e8409dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47008/0 (socket says 172.21.15.103:47008) 2023-07-02T09:01:11.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e827fc700 1 -- 172.21.15.103:0/4272788942 learned_addr learned my addr 172.21.15.103:0/4272788942 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:11.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e82ffd700 1 -- 172.21.15.103:0/4272788942 >> v1:172.21.15.103:6789/0 conn(0x7f1e8409e1f0 legacy=0x7f1e84132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:11.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e81ffb700 1 -- 172.21.15.103:0/4272788942 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564925222 0 0) 0x7f1e8409f240 con 0x7f1e8409e1f0 2023-07-02T09:01:11.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e82ffd700 1 -- 172.21.15.103:0/4272788942 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e840722d0 con 0x7f1e8409d8a0 2023-07-02T09:01:11.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e82ffd700 1 --2- 172.21.15.103:0/4272788942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e8409dcb0 crc :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854a2424f7efb5f6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:11.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e81ffb700 1 -- 172.21.15.103:0/4272788942 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1e78009cd0 con 0x7f1e8409d8a0 2023-07-02T09:01:11.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e81ffb700 1 -- 172.21.15.103:0/4272788942 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1e78009e30 con 0x7f1e8409d8a0 2023-07-02T09:01:11.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.825+0000 7f1e81ffb700 1 -- 172.21.15.103:0/4272788942 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1e7800e360 con 0x7f1e8409d8a0 2023-07-02T09:01:11.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.826+0000 7f1e89788700 1 -- 172.21.15.103:0/4272788942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 msgr2=0x7f1e8409dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:11.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.826+0000 7f1e89788700 1 --2- 172.21.15.103:0/4272788942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e8409dcb0 crc :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:11.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.826+0000 7f1e89788700 1 -- 172.21.15.103:0/4272788942 shutdown_connections 2023-07-02T09:01:11.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.826+0000 7f1e89788700 1 --2- 172.21.15.103:0/4272788942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e8409dcb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:11.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.826+0000 7f1e89788700 1 -- 172.21.15.103:0/4272788942 >> 172.21.15.103:0/4272788942 conn(0x7f1e840a3470 msgr2=0x7f1e840a3880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:11.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.826+0000 7f1e89788700 1 -- 172.21.15.103:0/4272788942 shutdown_connections 2023-07-02T09:01:11.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.826+0000 7f1e89788700 1 -- 172.21.15.103:0/4272788942 wait complete. 2023-07-02T09:01:11.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.826+0000 7f1e89788700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:11.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.827+0000 7f1e89788700 1 Processor -- start 2023-07-02T09:01:11.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.827+0000 7f1e89788700 1 -- start start 2023-07-02T09:01:11.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.828+0000 7f1e89788700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e841c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:11.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.828+0000 7f1e89788700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84124110 con 0x7f1e8409d8a0 2023-07-02T09:01:11.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.828+0000 7f1e82ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e841c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:11.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.828+0000 7f1e82ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e841c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47014/0 (socket says 172.21.15.103:47014) 2023-07-02T09:01:11.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.828+0000 7f1e82ffd700 1 -- 172.21.15.103:0/1985282443 learned_addr learned my addr 172.21.15.103:0/1985282443 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:11.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.828+0000 7f1e82ffd700 1 -- 172.21.15.103:0/1985282443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84122460 con 0x7f1e8409d8a0 2023-07-02T09:01:11.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.829+0000 7f1e82ffd700 1 --2- 172.21.15.103:0/1985282443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e841c6de0 crc :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:11.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.829+0000 7f1e6b7fe700 1 -- 172.21.15.103:0/1985282443 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1e78013530 con 0x7f1e8409d8a0 2023-07-02T09:01:11.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.829+0000 7f1e6b7fe700 1 -- 172.21.15.103:0/1985282443 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1e78013690 con 0x7f1e8409d8a0 2023-07-02T09:01:11.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.829+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e841220c0 con 0x7f1e8409d8a0 2023-07-02T09:01:11.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.829+0000 7f1e6b7fe700 1 -- 172.21.15.103:0/1985282443 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1e78013930 con 0x7f1e8409d8a0 2023-07-02T09:01:11.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.829+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e84125cd0 con 0x7f1e8409d8a0 2023-07-02T09:01:11.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.829+0000 7f1e6b7fe700 1 -- 172.21.15.103:0/1985282443 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f1e78013ae0 con 0x7f1e8409d8a0 2023-07-02T09:01:11.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.830+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e84124590 con 0x7f1e8409d8a0 2023-07-02T09:01:11.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.830+0000 7f1e6b7fe700 1 --2- 172.21.15.103:0/1985282443 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e700377b0 0x7f1e70039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:11.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.830+0000 7f1e6b7fe700 1 -- 172.21.15.103:0/1985282443 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f1e78018040 con 0x7f1e8409d8a0 2023-07-02T09:01:11.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.830+0000 7f1e827fc700 1 --2- 172.21.15.103:0/1985282443 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e700377b0 0x7f1e70039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:11.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.830+0000 7f1e827fc700 1 --2- 172.21.15.103:0/1985282443 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e700377b0 0x7f1e70039cc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:11.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:11.834+0000 7f1e6b7fe700 1 -- 172.21.15.103:0/1985282443 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f1e78018e00 con 0x7f1e8409d8a0 2023-07-02T09:01:12.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.025+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1e84061ea0 con 0x7f1e8409d8a0 2023-07-02T09:01:12.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.026+0000 7f1e6b7fe700 1 -- 172.21.15.103:0/1985282443 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f1e78010e60 con 0x7f1e8409d8a0 2023-07-02T09:01:12.026 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e700377b0 msgr2=0x7f1e70039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 --2- 172.21.15.103:0/1985282443 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e700377b0 0x7f1e70039cc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 msgr2=0x7f1e841c6de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 --2- 172.21.15.103:0/1985282443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e841c6de0 crc :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:12.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 shutdown_connections 2023-07-02T09:01:12.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 --2- 172.21.15.103:0/1985282443 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e700377b0 0x7f1e70039cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:12.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 --2- 172.21.15.103:0/1985282443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e8409d8a0 0x7f1e841c6de0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:12.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 >> 172.21.15.103:0/1985282443 conn(0x7f1e840a3470 msgr2=0x7f1e8406f740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:12.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 shutdown_connections 2023-07-02T09:01:12.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:12.028+0000 7f1e89788700 1 -- 172.21.15.103:0/1985282443 wait complete. 2023-07-02T09:01:12.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:13.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.181+0000 7f7659a15700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.186+0000 7f7659a15700 1 Processor -- start 2023-07-02T09:01:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.186+0000 7f7659a15700 1 -- start start 2023-07-02T09:01:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7659a15700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f765406f720 0x7f765406fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7659a15700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7654067780 con 0x7f7654070070 2023-07-02T09:01:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7659a15700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7654067f80 con 0x7f765406f720 2023-07-02T09:01:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7652ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f765406f720 0x7f765406fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f76527fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7654070070 0x7f76541307f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36112/0 (socket says 172.21.15.103:36112) 2023-07-02T09:01:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7652ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f765406f720 0x7f765406fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47018/0 (socket says 172.21.15.103:47018) 2023-07-02T09:01:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f76527fc700 1 -- 172.21.15.103:0/429921337 learned_addr learned my addr 172.21.15.103:0/429921337 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7652ffd700 1 -- 172.21.15.103:0/429921337 >> v1:172.21.15.103:6789/0 conn(0x7f7654070070 legacy=0x7f76541307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7652ffd700 1 -- 172.21.15.103:0/429921337 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654093f30 con 0x7f765406f720 2023-07-02T09:01:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7651ffb700 1 -- 172.21.15.103:0/429921337 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032211619 0 0) 0x7f7654067780 con 0x7f7654070070 2023-07-02T09:01:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7652ffd700 1 --2- 172.21.15.103:0/429921337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f765406fb30 crc :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3397f60dad2db0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7651ffb700 1 -- 172.21.15.103:0/429921337 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7648009cd0 con 0x7f765406f720 2023-07-02T09:01:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.187+0000 7f7651ffb700 1 -- 172.21.15.103:0/429921337 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7648009e30 con 0x7f765406f720 2023-07-02T09:01:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7651ffb700 1 -- 172.21.15.103:0/429921337 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f764800e360 con 0x7f765406f720 2023-07-02T09:01:13.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7659a15700 1 -- 172.21.15.103:0/429921337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 msgr2=0x7f765406fb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:13.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7659a15700 1 --2- 172.21.15.103:0/429921337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f765406fb30 crc :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:13.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7659a15700 1 -- 172.21.15.103:0/429921337 shutdown_connections 2023-07-02T09:01:13.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7659a15700 1 --2- 172.21.15.103:0/429921337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f765406fb30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:13.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7659a15700 1 -- 172.21.15.103:0/429921337 >> 172.21.15.103:0/429921337 conn(0x7f76540a22a0 msgr2=0x7f765406d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7659a15700 1 -- 172.21.15.103:0/429921337 shutdown_connections 2023-07-02T09:01:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7659a15700 1 -- 172.21.15.103:0/429921337 wait complete. 2023-07-02T09:01:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.188+0000 7f7659a15700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.189+0000 7f7659a15700 1 Processor -- start 2023-07-02T09:01:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.190+0000 7f7659a15700 1 -- start start 2023-07-02T09:01:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.190+0000 7f7659a15700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f76540708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.190+0000 7f7659a15700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765409d220 con 0x7f765406f720 2023-07-02T09:01:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.190+0000 7f7652ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f76540708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.190+0000 7f7652ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f76540708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47024/0 (socket says 172.21.15.103:47024) 2023-07-02T09:01:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.190+0000 7f7652ffd700 1 -- 172.21.15.103:0/56967706 learned_addr learned my addr 172.21.15.103:0/56967706 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.191+0000 7f7652ffd700 1 -- 172.21.15.103:0/56967706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654131250 con 0x7f765406f720 2023-07-02T09:01:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.191+0000 7f7652ffd700 1 --2- 172.21.15.103:0/56967706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f76540708b0 crc :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.191+0000 7f763bfff700 1 -- 172.21.15.103:0/56967706 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7648013530 con 0x7f765406f720 2023-07-02T09:01:13.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.191+0000 7f763bfff700 1 -- 172.21.15.103:0/56967706 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7648013690 con 0x7f765406f720 2023-07-02T09:01:13.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.191+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765409d540 con 0x7f765406f720 2023-07-02T09:01:13.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.191+0000 7f763bfff700 1 -- 172.21.15.103:0/56967706 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7648013930 con 0x7f765406f720 2023-07-02T09:01:13.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.191+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76541314b0 con 0x7f765406f720 2023-07-02T09:01:13.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.192+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7654061ea0 con 0x7f765406f720 2023-07-02T09:01:13.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.192+0000 7f763bfff700 1 -- 172.21.15.103:0/56967706 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7648013ae0 con 0x7f765406f720 2023-07-02T09:01:13.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.192+0000 7f763bfff700 1 --2- 172.21.15.103:0/56967706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f76400377b0 0x7f7640039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:13.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.192+0000 7f763bfff700 1 -- 172.21.15.103:0/56967706 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7648018040 con 0x7f765406f720 2023-07-02T09:01:13.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.193+0000 7f76527fc700 1 --2- 172.21.15.103:0/56967706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f76400377b0 0x7f7640039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:13.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.193+0000 7f76527fc700 1 --2- 172.21.15.103:0/56967706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f76400377b0 0x7f7640039cc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:13.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.197+0000 7f763bfff700 1 -- 172.21.15.103:0/56967706 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7648018e00 con 0x7f765406f720 2023-07-02T09:01:13.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.385+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7654073ca0 con 0x7f765406f720 2023-07-02T09:01:13.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.386+0000 7f763bfff700 1 -- 172.21.15.103:0/56967706 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7648010e60 con 0x7f765406f720 2023-07-02T09:01:13.386 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:13.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.387+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f76400377b0 msgr2=0x7f7640039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:13.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 --2- 172.21.15.103:0/56967706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f76400377b0 0x7f7640039cc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:13.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 msgr2=0x7f76540708b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:13.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 --2- 172.21.15.103:0/56967706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f76540708b0 crc :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:13.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 shutdown_connections 2023-07-02T09:01:13.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 --2- 172.21.15.103:0/56967706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f76400377b0 0x7f7640039cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:13.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 --2- 172.21.15.103:0/56967706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f765406f720 0x7f76540708b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:13.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 >> 172.21.15.103:0/56967706 conn(0x7f76540a22a0 msgr2=0x7f765408f670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:13.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 shutdown_connections 2023-07-02T09:01:13.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:13.388+0000 7f7659a15700 1 -- 172.21.15.103:0/56967706 wait complete. 2023-07-02T09:01:13.401 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:13.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~88s 2023-07-02T09:01:14.401 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:14.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.539+0000 7f31a5dc9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:14.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.544+0000 7f31a5dc9700 1 Processor -- start 2023-07-02T09:01:14.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.544+0000 7f31a5dc9700 1 -- start start 2023-07-02T09:01:14.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f31a5dc9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f31a0133360 0x7f31a009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:14.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f31a5dc9700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a009e260 con 0x7f31a0132f50 2023-07-02T09:01:14.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f31a5dc9700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a00a1640 con 0x7f31a0133360 2023-07-02T09:01:14.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f319effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f31a0133360 0x7f31a009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:14.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f319f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f31a0132f50 0x7f31a009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40616/0 (socket says 172.21.15.103:40616) 2023-07-02T09:01:14.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f319effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f31a0133360 0x7f31a009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41118/0 (socket says 172.21.15.103:41118) 2023-07-02T09:01:14.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f319f7fe700 1 -- 172.21.15.103:0/3531287947 learned_addr learned my addr 172.21.15.103:0/3531287947 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:14.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f319effd700 1 -- 172.21.15.103:0/3531287947 >> v1:172.21.15.103:6789/0 conn(0x7f31a0132f50 legacy=0x7f31a009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:14.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f319effd700 1 -- 172.21.15.103:0/3531287947 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a0132a20 con 0x7f31a0133360 2023-07-02T09:01:14.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f319effd700 1 --2- 172.21.15.103:0/3531287947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0133360 0x7f31a009cc90 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eaa6eeef40abbf8 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:14.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.545+0000 7f319e7fc700 1 -- 172.21.15.103:0/3531287947 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3190009cd0 con 0x7f31a0133360 2023-07-02T09:01:14.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f319e7fc700 1 -- 172.21.15.103:0/3531287947 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3190009e30 con 0x7f31a0133360 2023-07-02T09:01:14.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f319e7fc700 1 -- 172.21.15.103:0/3531287947 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f319000e360 con 0x7f31a0133360 2023-07-02T09:01:14.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3531287947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0133360 msgr2=0x7f31a009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f31a5dc9700 1 --2- 172.21.15.103:0/3531287947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0133360 0x7f31a009cc90 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3531287947 shutdown_connections 2023-07-02T09:01:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f31a5dc9700 1 --2- 172.21.15.103:0/3531287947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0133360 0x7f31a009cc90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3531287947 >> 172.21.15.103:0/3531287947 conn(0x7f31a00a2080 msgr2=0x7f31a00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3531287947 shutdown_connections 2023-07-02T09:01:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.546+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3531287947 wait complete. 2023-07-02T09:01:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.547+0000 7f31a5dc9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.547+0000 7f31a5dc9700 1 Processor -- start 2023-07-02T09:01:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.548+0000 7f31a5dc9700 1 -- start start 2023-07-02T09:01:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.548+0000 7f31a5dc9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0132f50 0x7f31a012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.548+0000 7f31a5dc9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a006d260 con 0x7f31a0132f50 2023-07-02T09:01:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.548+0000 7f319f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0132f50 0x7f31a012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.548+0000 7f319f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0132f50 0x7f31a012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41126/0 (socket says 172.21.15.103:41126) 2023-07-02T09:01:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.548+0000 7f319f7fe700 1 -- 172.21.15.103:0/3655893926 learned_addr learned my addr 172.21.15.103:0/3655893926 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.549+0000 7f319f7fe700 1 -- 172.21.15.103:0/3655893926 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a012bb90 con 0x7f31a0132f50 2023-07-02T09:01:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.549+0000 7f319f7fe700 1 --2- 172.21.15.103:0/3655893926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0132f50 0x7f31a012bea0 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.549+0000 7f3197fff700 1 -- 172.21.15.103:0/3655893926 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3188005a60 con 0x7f31a0132f50 2023-07-02T09:01:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.549+0000 7f3197fff700 1 -- 172.21.15.103:0/3655893926 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3188005bc0 con 0x7f31a0132f50 2023-07-02T09:01:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.549+0000 7f3197fff700 1 -- 172.21.15.103:0/3655893926 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3188005e60 con 0x7f31a0132f50 2023-07-02T09:01:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.549+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a006f730 con 0x7f31a0132f50 2023-07-02T09:01:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.549+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a006f490 con 0x7f31a0132f50 2023-07-02T09:01:14.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.550+0000 7f3197fff700 1 -- 172.21.15.103:0/3655893926 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f31880070b0 con 0x7f31a0132f50 2023-07-02T09:01:14.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.550+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a0061ea0 con 0x7f31a0132f50 2023-07-02T09:01:14.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.550+0000 7f3197fff700 1 --2- 172.21.15.103:0/3655893926 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31800377c0 0x7f3180039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:14.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.550+0000 7f3197fff700 1 -- 172.21.15.103:0/3655893926 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f31880469f0 con 0x7f31a0132f50 2023-07-02T09:01:14.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.550+0000 7f319effd700 1 --2- 172.21.15.103:0/3655893926 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31800377c0 0x7f3180039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:14.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.551+0000 7f319effd700 1 --2- 172.21.15.103:0/3655893926 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31800377c0 0x7f3180039cd0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:14.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.555+0000 7f3197fff700 1 -- 172.21.15.103:0/3655893926 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f318800a4a0 con 0x7f31a0132f50 2023-07-02T09:01:14.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.742+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f31a006f490 con 0x7f31a0132f50 2023-07-02T09:01:14.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.743+0000 7f3197fff700 1 -- 172.21.15.103:0/3655893926 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f31a006f490 con 0x7f31a0132f50 2023-07-02T09:01:14.743 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:14.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31800377c0 msgr2=0x7f3180039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:14.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 --2- 172.21.15.103:0/3655893926 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31800377c0 0x7f3180039cd0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:14.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0132f50 msgr2=0x7f31a012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:14.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 --2- 172.21.15.103:0/3655893926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0132f50 0x7f31a012bea0 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:14.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 shutdown_connections 2023-07-02T09:01:14.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 --2- 172.21.15.103:0/3655893926 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31800377c0 0x7f3180039cd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:14.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 --2- 172.21.15.103:0/3655893926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a0132f50 0x7f31a012bea0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:14.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 >> 172.21.15.103:0/3655893926 conn(0x7f31a00a2080 msgr2=0x7f31a01bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:14.747 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 shutdown_connections 2023-07-02T09:01:14.747 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:14.745+0000 7f31a5dc9700 1 -- 172.21.15.103:0/3655893926 wait complete. 2023-07-02T09:01:14.758 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:15.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:15.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.892+0000 7f07eaf79700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:15.897 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.897+0000 7f07eaf79700 1 Processor -- start 2023-07-02T09:01:15.898 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.897+0000 7f07eaf79700 1 -- start start 2023-07-02T09:01:15.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07eaf79700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f07e4133360 0x7f07e409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:15.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07eaf79700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e409e260 con 0x7f07e4132f50 2023-07-02T09:01:15.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07eaf79700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e40a1640 con 0x7f07e4133360 2023-07-02T09:01:15.899 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e3fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f07e4133360 0x7f07e409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e8d15700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f07e4132f50 0x7f07e409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40632/0 (socket says 172.21.15.103:40632) 2023-07-02T09:01:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e8d15700 1 -- 172.21.15.103:0/1723745877 learned_addr learned my addr 172.21.15.103:0/1723745877 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e3fff700 1 -- 172.21.15.103:0/1723745877 >> v1:172.21.15.103:6789/0 conn(0x7f07e4132f50 legacy=0x7f07e409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e37fe700 1 -- 172.21.15.103:0/1723745877 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941360397 0 0) 0x7f07e409e260 con 0x7f07e4132f50 2023-07-02T09:01:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e3fff700 1 -- 172.21.15.103:0/1723745877 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e4132a20 con 0x7f07e4133360 2023-07-02T09:01:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e3fff700 1 --2- 172.21.15.103:0/1723745877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4133360 0x7f07e409cc90 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c312d1f704e56efc server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e37fe700 1 -- 172.21.15.103:0/1723745877 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f07d0009cd0 con 0x7f07e4133360 2023-07-02T09:01:15.901 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e37fe700 1 -- 172.21.15.103:0/1723745877 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f07d0009e30 con 0x7f07e4133360 2023-07-02T09:01:15.901 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.898+0000 7f07e37fe700 1 -- 172.21.15.103:0/1723745877 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f07d000e360 con 0x7f07e4133360 2023-07-02T09:01:15.902 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.899+0000 7f07eaf79700 1 -- 172.21.15.103:0/1723745877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4133360 msgr2=0x7f07e409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:15.903 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.899+0000 7f07eaf79700 1 --2- 172.21.15.103:0/1723745877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4133360 0x7f07e409cc90 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:15.903 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.899+0000 7f07eaf79700 1 -- 172.21.15.103:0/1723745877 shutdown_connections 2023-07-02T09:01:15.903 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.899+0000 7f07eaf79700 1 --2- 172.21.15.103:0/1723745877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4133360 0x7f07e409cc90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:15.903 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.899+0000 7f07eaf79700 1 -- 172.21.15.103:0/1723745877 >> 172.21.15.103:0/1723745877 conn(0x7f07e40a2080 msgr2=0x7f07e40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:15.903 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.899+0000 7f07eaf79700 1 -- 172.21.15.103:0/1723745877 shutdown_connections 2023-07-02T09:01:15.903 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.899+0000 7f07eaf79700 1 -- 172.21.15.103:0/1723745877 wait complete. 2023-07-02T09:01:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.899+0000 7f07eaf79700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.900+0000 7f07eaf79700 1 Processor -- start 2023-07-02T09:01:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.900+0000 7f07eaf79700 1 -- start start 2023-07-02T09:01:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.901+0000 7f07eaf79700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4132f50 0x7f07e412c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.901+0000 7f07eaf79700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e406d2f0 con 0x7f07e4132f50 2023-07-02T09:01:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.901+0000 7f07e8d15700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4132f50 0x7f07e412c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.901+0000 7f07e8d15700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4132f50 0x7f07e412c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41146/0 (socket says 172.21.15.103:41146) 2023-07-02T09:01:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.901+0000 7f07e8d15700 1 -- 172.21.15.103:0/2565106097 learned_addr learned my addr 172.21.15.103:0/2565106097 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.901+0000 7f07e8d15700 1 -- 172.21.15.103:0/2565106097 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e406dd60 con 0x7f07e4132f50 2023-07-02T09:01:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.901+0000 7f07e8d15700 1 --2- 172.21.15.103:0/2565106097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4132f50 0x7f07e412c1a0 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.902+0000 7f07e17fa700 1 -- 172.21.15.103:0/2565106097 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f07d8004b80 con 0x7f07e4132f50 2023-07-02T09:01:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.902+0000 7f07e17fa700 1 -- 172.21.15.103:0/2565106097 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f07d8004ce0 con 0x7f07e4132f50 2023-07-02T09:01:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.902+0000 7f07e17fa700 1 -- 172.21.15.103:0/2565106097 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f07d800c4f0 con 0x7f07e4132f50 2023-07-02T09:01:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.902+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e412bc50 con 0x7f07e4132f50 2023-07-02T09:01:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.902+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e406f850 con 0x7f07e4132f50 2023-07-02T09:01:15.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.903+0000 7f07e17fa700 1 -- 172.21.15.103:0/2565106097 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f07d800c650 con 0x7f07e4132f50 2023-07-02T09:01:15.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.903+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e406dbb0 con 0x7f07e4132f50 2023-07-02T09:01:15.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.903+0000 7f07e17fa700 1 --2- 172.21.15.103:0/2565106097 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f07c80377c0 0x7f07c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:15.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.903+0000 7f07e17fa700 1 -- 172.21.15.103:0/2565106097 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f07d8046970 con 0x7f07e4132f50 2023-07-02T09:01:15.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.903+0000 7f07e3fff700 1 --2- 172.21.15.103:0/2565106097 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f07c80377c0 0x7f07c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:15.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.904+0000 7f07e3fff700 1 --2- 172.21.15.103:0/2565106097 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f07c80377c0 0x7f07c8039cd0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:15.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:15.907+0000 7f07e17fa700 1 -- 172.21.15.103:0/2565106097 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f07d8009c80 con 0x7f07e4132f50 2023-07-02T09:01:16.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.095+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f07e4061ea0 con 0x7f07e4132f50 2023-07-02T09:01:16.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.096+0000 7f07e17fa700 1 -- 172.21.15.103:0/2565106097 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f07d80098a0 con 0x7f07e4132f50 2023-07-02T09:01:16.097 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:16.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f07c80377c0 msgr2=0x7f07c8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:16.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 --2- 172.21.15.103:0/2565106097 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f07c80377c0 0x7f07c8039cd0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:16.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4132f50 msgr2=0x7f07e412c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:16.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 --2- 172.21.15.103:0/2565106097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4132f50 0x7f07e412c1a0 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:16.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 shutdown_connections 2023-07-02T09:01:16.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 --2- 172.21.15.103:0/2565106097 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f07c80377c0 0x7f07c8039cd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:16.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 --2- 172.21.15.103:0/2565106097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07e4132f50 0x7f07e412c1a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:16.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 >> 172.21.15.103:0/2565106097 conn(0x7f07e40a2080 msgr2=0x7f07e4130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:16.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 shutdown_connections 2023-07-02T09:01:16.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:16.098+0000 7f07eaf79700 1 -- 172.21.15.103:0/2565106097 wait complete. 2023-07-02T09:01:16.111 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:17.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:17.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.250+0000 7f11e4c2a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:17.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.255+0000 7f11e4c2a700 1 Processor -- start 2023-07-02T09:01:17.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.255+0000 7f11e4c2a700 1 -- start start 2023-07-02T09:01:17.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11e4c2a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11e006d6e0 0x7f11e0099b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:17.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11e4c2a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e009cea0 con 0x7f11e009a050 2023-07-02T09:01:17.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11e4c2a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e006d490 con 0x7f11e006d6e0 2023-07-02T09:01:17.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11de59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11e006d6e0 0x7f11e0099b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:17.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11ddd9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f11e009a050 0x7f11e012f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40644/0 (socket says 172.21.15.103:40644) 2023-07-02T09:01:17.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11de59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11e006d6e0 0x7f11e0099b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41154/0 (socket says 172.21.15.103:41154) 2023-07-02T09:01:17.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11ddd9b700 1 -- 172.21.15.103:0/4164033322 learned_addr learned my addr 172.21.15.103:0/4164033322 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:17.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11de59c700 1 -- 172.21.15.103:0/4164033322 >> v1:172.21.15.103:6789/0 conn(0x7f11e009a050 legacy=0x7f11e012f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:17.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11de59c700 1 -- 172.21.15.103:0/4164033322 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e006dc80 con 0x7f11e006d6e0 2023-07-02T09:01:17.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11de59c700 1 --2- 172.21.15.103:0/4164033322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e0099b10 crc :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a8e8c8d3f3807d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:17.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11dd59a700 1 -- 172.21.15.103:0/4164033322 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f11c8009cd0 con 0x7f11e006d6e0 2023-07-02T09:01:17.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11dd59a700 1 -- 172.21.15.103:0/4164033322 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f11c8009e30 con 0x7f11e006d6e0 2023-07-02T09:01:17.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.256+0000 7f11dd59a700 1 -- 172.21.15.103:0/4164033322 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f11c800e360 con 0x7f11e006d6e0 2023-07-02T09:01:17.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.257+0000 7f11e4c2a700 1 -- 172.21.15.103:0/4164033322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 msgr2=0x7f11e0099b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:17.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.257+0000 7f11e4c2a700 1 --2- 172.21.15.103:0/4164033322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e0099b10 crc :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:17.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.257+0000 7f11e4c2a700 1 -- 172.21.15.103:0/4164033322 shutdown_connections 2023-07-02T09:01:17.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.257+0000 7f11e4c2a700 1 --2- 172.21.15.103:0/4164033322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e0099b10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:17.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.257+0000 7f11e4c2a700 1 -- 172.21.15.103:0/4164033322 >> 172.21.15.103:0/4164033322 conn(0x7f11e009d810 msgr2=0x7f11e009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:17.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.257+0000 7f11e4c2a700 1 -- 172.21.15.103:0/4164033322 shutdown_connections 2023-07-02T09:01:17.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.257+0000 7f11e4c2a700 1 -- 172.21.15.103:0/4164033322 wait complete. 2023-07-02T09:01:17.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.257+0000 7f11e4c2a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:17.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.258+0000 7f11e4c2a700 1 Processor -- start 2023-07-02T09:01:17.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.258+0000 7f11e4c2a700 1 -- start start 2023-07-02T09:01:17.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.259+0000 7f11e4c2a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e011dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:17.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.259+0000 7f11e4c2a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e011f130 con 0x7f11e006d6e0 2023-07-02T09:01:17.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.259+0000 7f11de59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e011dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:17.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.259+0000 7f11de59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e011dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41162/0 (socket says 172.21.15.103:41162) 2023-07-02T09:01:17.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.259+0000 7f11de59c700 1 -- 172.21.15.103:0/1789714323 learned_addr learned my addr 172.21.15.103:0/1789714323 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:17.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.259+0000 7f11de59c700 1 -- 172.21.15.103:0/1789714323 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e0124530 con 0x7f11e006d6e0 2023-07-02T09:01:17.262 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.260+0000 7f11de59c700 1 --2- 172.21.15.103:0/1789714323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e011dbe0 crc :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:17.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.260+0000 7f11d6ffd700 1 -- 172.21.15.103:0/1789714323 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f11c8013740 con 0x7f11e006d6e0 2023-07-02T09:01:17.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.260+0000 7f11d6ffd700 1 -- 172.21.15.103:0/1789714323 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f11c80138a0 con 0x7f11e006d6e0 2023-07-02T09:01:17.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.260+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e0124ff0 con 0x7f11e006d6e0 2023-07-02T09:01:17.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.260+0000 7f11d6ffd700 1 -- 172.21.15.103:0/1789714323 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f11c8013b40 con 0x7f11e006d6e0 2023-07-02T09:01:17.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.260+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e01241d0 con 0x7f11e006d6e0 2023-07-02T09:01:17.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.261+0000 7f11d6ffd700 1 -- 172.21.15.103:0/1789714323 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f11c8013cf0 con 0x7f11e006d6e0 2023-07-02T09:01:17.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.261+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e0061ea0 con 0x7f11e006d6e0 2023-07-02T09:01:17.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.261+0000 7f11d6ffd700 1 --2- 172.21.15.103:0/1789714323 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377c0 0x7f11c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:17.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.261+0000 7f11d6ffd700 1 -- 172.21.15.103:0/1789714323 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f11c8018040 con 0x7f11e006d6e0 2023-07-02T09:01:17.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.261+0000 7f11ddd9b700 1 --2- 172.21.15.103:0/1789714323 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377c0 0x7f11c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:17.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.262+0000 7f11ddd9b700 1 --2- 172.21.15.103:0/1789714323 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377c0 0x7f11c0039cd0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:17.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.266+0000 7f11d6ffd700 1 -- 172.21.15.103:0/1789714323 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f11c80189b0 con 0x7f11e006d6e0 2023-07-02T09:01:17.455 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.455+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f11e0073ca0 con 0x7f11e006d6e0 2023-07-02T09:01:17.455 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.455+0000 7f11d6ffd700 1 -- 172.21.15.103:0/1789714323 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f11c800f2a0 con 0x7f11e006d6e0 2023-07-02T09:01:17.456 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:17.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377c0 msgr2=0x7f11c0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:17.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 --2- 172.21.15.103:0/1789714323 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377c0 0x7f11c0039cd0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:17.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 msgr2=0x7f11e011dbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:17.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 --2- 172.21.15.103:0/1789714323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e011dbe0 crc :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:17.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 shutdown_connections 2023-07-02T09:01:17.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 --2- 172.21.15.103:0/1789714323 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11c00377c0 0x7f11c0039cd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:17.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 --2- 172.21.15.103:0/1789714323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11e006d6e0 0x7f11e011dbe0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:17.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 >> 172.21.15.103:0/1789714323 conn(0x7f11e009d810 msgr2=0x7f11e0117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:17.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 shutdown_connections 2023-07-02T09:01:17.459 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:17.457+0000 7f11e4c2a700 1 -- 172.21.15.103:0/1789714323 wait complete. 2023-07-02T09:01:17.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:18.470 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:18.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.609+0000 7f6d3d0c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:18.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.614+0000 7f6d3d0c3700 1 Processor -- start 2023-07-02T09:01:18.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.614+0000 7f6d3d0c3700 1 -- start start 2023-07-02T09:01:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.615+0000 7f6d3d0c3700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6d38132030 0x7f6d38134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.615+0000 7f6d3d0c3700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3806df60 con 0x7f6d3812f250 2023-07-02T09:01:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.615+0000 7f6d3d0c3700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3809a710 con 0x7f6d38132030 2023-07-02T09:01:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.615+0000 7f6d3659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6d38132030 0x7f6d38134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.615+0000 7f6d36d9d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6d3812f250 0x7f6d3812f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40658/0 (socket says 172.21.15.103:40658) 2023-07-02T09:01:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.615+0000 7f6d36d9d700 1 -- 172.21.15.103:0/2223259394 learned_addr learned my addr 172.21.15.103:0/2223259394 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d3659c700 1 -- 172.21.15.103:0/2223259394 >> v1:172.21.15.103:6789/0 conn(0x7f6d3812f250 legacy=0x7f6d3812f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d3659c700 1 -- 172.21.15.103:0/2223259394 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3812ed20 con 0x7f6d38132030 2023-07-02T09:01:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d3659c700 1 --2- 172.21.15.103:0/2223259394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d38132030 0x7f6d38134450 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926f32823affc2f9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d35d9b700 1 -- 172.21.15.103:0/2223259394 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6d2800ac60 con 0x7f6d38132030 2023-07-02T09:01:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d35d9b700 1 -- 172.21.15.103:0/2223259394 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6d2800adc0 con 0x7f6d38132030 2023-07-02T09:01:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d35d9b700 1 -- 172.21.15.103:0/2223259394 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6d2800f470 con 0x7f6d38132030 2023-07-02T09:01:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/2223259394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d38132030 msgr2=0x7f6d38134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d3d0c3700 1 --2- 172.21.15.103:0/2223259394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d38132030 0x7f6d38134450 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/2223259394 shutdown_connections 2023-07-02T09:01:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d3d0c3700 1 --2- 172.21.15.103:0/2223259394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d38132030 0x7f6d38134450 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.616+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/2223259394 >> 172.21.15.103:0/2223259394 conn(0x7f6d3809d810 msgr2=0x7f6d3809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.617+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/2223259394 shutdown_connections 2023-07-02T09:01:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.617+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/2223259394 wait complete. 2023-07-02T09:01:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.617+0000 7f6d3d0c3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.618+0000 7f6d3d0c3700 1 Processor -- start 2023-07-02T09:01:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.618+0000 7f6d3d0c3700 1 -- start start 2023-07-02T09:01:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.619+0000 7f6d3d0c3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d3812f250 0x7f6d38070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.619+0000 7f6d3d0c3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d38071240 con 0x7f6d3812f250 2023-07-02T09:01:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.619+0000 7f6d36d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d3812f250 0x7f6d38070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.619+0000 7f6d36d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d3812f250 0x7f6d38070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41184/0 (socket says 172.21.15.103:41184) 2023-07-02T09:01:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.619+0000 7f6d36d9d700 1 -- 172.21.15.103:0/4151713276 learned_addr learned my addr 172.21.15.103:0/4151713276 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.620+0000 7f6d36d9d700 1 -- 172.21.15.103:0/4151713276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d38119220 con 0x7f6d3812f250 2023-07-02T09:01:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.620+0000 7f6d36d9d700 1 --2- 172.21.15.103:0/4151713276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d3812f250 0x7f6d38070100 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.620+0000 7f6d1f7fe700 1 -- 172.21.15.103:0/4151713276 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6d2c00ec60 con 0x7f6d3812f250 2023-07-02T09:01:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.620+0000 7f6d1f7fe700 1 -- 172.21.15.103:0/4151713276 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6d2c00edc0 con 0x7f6d3812f250 2023-07-02T09:01:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.620+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d38071d60 con 0x7f6d3812f250 2023-07-02T09:01:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.620+0000 7f6d1f7fe700 1 -- 172.21.15.103:0/4151713276 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6d2c005a60 con 0x7f6d3812f250 2023-07-02T09:01:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.620+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d38118d50 con 0x7f6d3812f250 2023-07-02T09:01:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.621+0000 7f6d1f7fe700 1 -- 172.21.15.103:0/4151713276 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6d2c005bc0 con 0x7f6d3812f250 2023-07-02T09:01:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.621+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d38061ea0 con 0x7f6d3812f250 2023-07-02T09:01:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.621+0000 7f6d1f7fe700 1 --2- 172.21.15.103:0/4151713276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d240377c0 0x7f6d24039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.621+0000 7f6d1f7fe700 1 -- 172.21.15.103:0/4151713276 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6d2c049610 con 0x7f6d3812f250 2023-07-02T09:01:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.621+0000 7f6d3659c700 1 --2- 172.21.15.103:0/4151713276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d240377c0 0x7f6d24039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.622+0000 7f6d3659c700 1 --2- 172.21.15.103:0/4151713276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d240377c0 0x7f6d24039cd0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:18.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.626+0000 7f6d1f7fe700 1 -- 172.21.15.103:0/4151713276 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6d2c011440 con 0x7f6d3812f250 2023-07-02T09:01:18.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.815+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6d3812c6c0 con 0x7f6d3812f250 2023-07-02T09:01:18.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.816+0000 7f6d1f7fe700 1 -- 172.21.15.103:0/4151713276 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6d2c011440 con 0x7f6d3812f250 2023-07-02T09:01:18.816 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d240377c0 msgr2=0x7f6d24039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 --2- 172.21.15.103:0/4151713276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d240377c0 0x7f6d24039cd0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d3812f250 msgr2=0x7f6d38070100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 --2- 172.21.15.103:0/4151713276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d3812f250 0x7f6d38070100 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:18.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 shutdown_connections 2023-07-02T09:01:18.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 --2- 172.21.15.103:0/4151713276 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6d240377c0 0x7f6d24039cd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:18.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 --2- 172.21.15.103:0/4151713276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d3812f250 0x7f6d38070100 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:18.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 >> 172.21.15.103:0/4151713276 conn(0x7f6d3809d810 msgr2=0x7f6d3812e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:18.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 shutdown_connections 2023-07-02T09:01:18.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:18.818+0000 7f6d3d0c3700 1 -- 172.21.15.103:0/4151713276 wait complete. 2023-07-02T09:01:18.831 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:19.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~94s 2023-07-02T09:01:19.832 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:19.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.972+0000 7f68e7487700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:19.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.977+0000 7f68e7487700 1 Processor -- start 2023-07-02T09:01:19.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.977+0000 7f68e7487700 1 -- start start 2023-07-02T09:01:19.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e7487700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68e0133360 0x7f68e009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:19.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e7487700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e009e260 con 0x7f68e0132f50 2023-07-02T09:01:19.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e7487700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e00a1640 con 0x7f68e0133360 2023-07-02T09:01:19.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e4a22700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68e0133360 0x7f68e009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:19.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e5223700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f68e0132f50 0x7f68e009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40660/0 (socket says 172.21.15.103:40660) 2023-07-02T09:01:19.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e5223700 1 -- 172.21.15.103:0/4118294675 learned_addr learned my addr 172.21.15.103:0/4118294675 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:19.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e4a22700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68e0133360 0x7f68e009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41188/0 (socket says 172.21.15.103:41188) 2023-07-02T09:01:19.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e4a22700 1 -- 172.21.15.103:0/4118294675 >> v1:172.21.15.103:6789/0 conn(0x7f68e0132f50 legacy=0x7f68e009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:19.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68e4a22700 1 -- 172.21.15.103:0/4118294675 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0132a20 con 0x7f68e0133360 2023-07-02T09:01:19.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.978+0000 7f68d7fff700 1 -- 172.21.15.103:0/4118294675 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683446832 0 0) 0x7f68e009e260 con 0x7f68e0132f50 2023-07-02T09:01:19.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68e4a22700 1 --2- 172.21.15.103:0/4118294675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0133360 0x7f68e009cc90 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473c07fbcf7150dd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:19.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68d7fff700 1 -- 172.21.15.103:0/4118294675 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f68dc009cd0 con 0x7f68e0133360 2023-07-02T09:01:19.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68d7fff700 1 -- 172.21.15.103:0/4118294675 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f68dc009e30 con 0x7f68e0133360 2023-07-02T09:01:19.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68d7fff700 1 -- 172.21.15.103:0/4118294675 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f68dc00e360 con 0x7f68e0133360 2023-07-02T09:01:19.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68e7487700 1 -- 172.21.15.103:0/4118294675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0133360 msgr2=0x7f68e009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:19.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68e7487700 1 --2- 172.21.15.103:0/4118294675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0133360 0x7f68e009cc90 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:19.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68e7487700 1 -- 172.21.15.103:0/4118294675 shutdown_connections 2023-07-02T09:01:19.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68e7487700 1 --2- 172.21.15.103:0/4118294675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0133360 0x7f68e009cc90 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:19.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68e7487700 1 -- 172.21.15.103:0/4118294675 >> 172.21.15.103:0/4118294675 conn(0x7f68e00a2080 msgr2=0x7f68e00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:19.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68e7487700 1 -- 172.21.15.103:0/4118294675 shutdown_connections 2023-07-02T09:01:19.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.979+0000 7f68e7487700 1 -- 172.21.15.103:0/4118294675 wait complete. 2023-07-02T09:01:19.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.980+0000 7f68e7487700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:19.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.980+0000 7f68e7487700 1 Processor -- start 2023-07-02T09:01:19.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.981+0000 7f68e7487700 1 -- start start 2023-07-02T09:01:19.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.981+0000 7f68e7487700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0132f50 0x7f68e01ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:19.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.981+0000 7f68e7487700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e006d370 con 0x7f68e0132f50 2023-07-02T09:01:19.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.981+0000 7f68e5223700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0132f50 0x7f68e01ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:19.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.981+0000 7f68e5223700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0132f50 0x7f68e01ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41194/0 (socket says 172.21.15.103:41194) 2023-07-02T09:01:19.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.981+0000 7f68e5223700 1 -- 172.21.15.103:0/144267825 learned_addr learned my addr 172.21.15.103:0/144267825 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:19.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.982+0000 7f68e5223700 1 -- 172.21.15.103:0/144267825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e006dde0 con 0x7f68e0132f50 2023-07-02T09:01:19.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.982+0000 7f68e5223700 1 --2- 172.21.15.103:0/144267825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0132f50 0x7f68e01ba9b0 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:19.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.982+0000 7f68d5ffb700 1 -- 172.21.15.103:0/144267825 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f68d0004b80 con 0x7f68e0132f50 2023-07-02T09:01:19.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.982+0000 7f68d5ffb700 1 -- 172.21.15.103:0/144267825 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f68d0004ce0 con 0x7f68e0132f50 2023-07-02T09:01:19.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.982+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e01ba4f0 con 0x7f68e0132f50 2023-07-02T09:01:19.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.982+0000 7f68d5ffb700 1 -- 172.21.15.103:0/144267825 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f68d000c4f0 con 0x7f68e0132f50 2023-07-02T09:01:19.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.982+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e0071e40 con 0x7f68e0132f50 2023-07-02T09:01:19.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.983+0000 7f68d5ffb700 1 -- 172.21.15.103:0/144267825 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f68d000c650 con 0x7f68e0132f50 2023-07-02T09:01:19.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.983+0000 7f68d5ffb700 1 --2- 172.21.15.103:0/144267825 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f68c40377c0 0x7f68c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:19.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.983+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e0061ea0 con 0x7f68e0132f50 2023-07-02T09:01:19.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.983+0000 7f68d5ffb700 1 -- 172.21.15.103:0/144267825 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f68d0046970 con 0x7f68e0132f50 2023-07-02T09:01:19.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.983+0000 7f68e4a22700 1 --2- 172.21.15.103:0/144267825 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f68c40377c0 0x7f68c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:19.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.984+0000 7f68e4a22700 1 --2- 172.21.15.103:0/144267825 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f68c40377c0 0x7f68c4039cd0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:19.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:19.988+0000 7f68d5ffb700 1 -- 172.21.15.103:0/144267825 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f68d0009c80 con 0x7f68e0132f50 2023-07-02T09:01:20.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.176+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f68e00725b0 con 0x7f68e0132f50 2023-07-02T09:01:20.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.177+0000 7f68d5ffb700 1 -- 172.21.15.103:0/144267825 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f68d002c020 con 0x7f68e0132f50 2023-07-02T09:01:20.177 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:20.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f68c40377c0 msgr2=0x7f68c4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:20.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 --2- 172.21.15.103:0/144267825 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f68c40377c0 0x7f68c4039cd0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:20.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0132f50 msgr2=0x7f68e01ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:20.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 --2- 172.21.15.103:0/144267825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0132f50 0x7f68e01ba9b0 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:20.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 shutdown_connections 2023-07-02T09:01:20.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 --2- 172.21.15.103:0/144267825 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f68c40377c0 0x7f68c4039cd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:20.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 --2- 172.21.15.103:0/144267825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68e0132f50 0x7f68e01ba9b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:20.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 >> 172.21.15.103:0/144267825 conn(0x7f68e00a2080 msgr2=0x7f68e00a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:20.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 shutdown_connections 2023-07-02T09:01:20.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:20.179+0000 7f68e7487700 1 -- 172.21.15.103:0/144267825 wait complete. 2023-07-02T09:01:20.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:21.193 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:21.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.331+0000 7fad72e8e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:21.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.336+0000 7fad72e8e700 1 Processor -- start 2023-07-02T09:01:21.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.336+0000 7fad72e8e700 1 -- start start 2023-07-02T09:01:21.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad72e8e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fad6c133360 0x7fad6c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:21.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad72e8e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6c09e260 con 0x7fad6c132f50 2023-07-02T09:01:21.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad72e8e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6c0a1640 con 0x7fad6c133360 2023-07-02T09:01:21.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad6bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fad6c133360 0x7fad6c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:21.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad70c2a700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fad6c132f50 0x7fad6c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40662/0 (socket says 172.21.15.103:40662) 2023-07-02T09:01:21.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad6bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fad6c133360 0x7fad6c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41200/0 (socket says 172.21.15.103:41200) 2023-07-02T09:01:21.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad70c2a700 1 -- 172.21.15.103:0/330240347 learned_addr learned my addr 172.21.15.103:0/330240347 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:21.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad6bfff700 1 -- 172.21.15.103:0/330240347 >> v1:172.21.15.103:6789/0 conn(0x7fad6c132f50 legacy=0x7fad6c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:21.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.337+0000 7fad6bfff700 1 -- 172.21.15.103:0/330240347 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c132a20 con 0x7fad6c133360 2023-07-02T09:01:21.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad6bfff700 1 --2- 172.21.15.103:0/330240347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c133360 0x7fad6c09cc90 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53f71229b3ec6b3 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:21.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad6b7fe700 1 -- 172.21.15.103:0/330240347 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fad58009cd0 con 0x7fad6c133360 2023-07-02T09:01:21.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad6b7fe700 1 -- 172.21.15.103:0/330240347 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fad58009e30 con 0x7fad6c133360 2023-07-02T09:01:21.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad6b7fe700 1 -- 172.21.15.103:0/330240347 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fad5800e360 con 0x7fad6c133360 2023-07-02T09:01:21.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad72e8e700 1 -- 172.21.15.103:0/330240347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c133360 msgr2=0x7fad6c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:21.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad72e8e700 1 --2- 172.21.15.103:0/330240347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c133360 0x7fad6c09cc90 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:21.342 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad72e8e700 1 -- 172.21.15.103:0/330240347 shutdown_connections 2023-07-02T09:01:21.342 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad72e8e700 1 --2- 172.21.15.103:0/330240347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c133360 0x7fad6c09cc90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:21.342 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad72e8e700 1 -- 172.21.15.103:0/330240347 >> 172.21.15.103:0/330240347 conn(0x7fad6c0a2080 msgr2=0x7fad6c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:21.342 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.338+0000 7fad72e8e700 1 -- 172.21.15.103:0/330240347 shutdown_connections 2023-07-02T09:01:21.342 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.339+0000 7fad72e8e700 1 -- 172.21.15.103:0/330240347 wait complete. 2023-07-02T09:01:21.342 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.339+0000 7fad72e8e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:21.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.340+0000 7fad72e8e700 1 Processor -- start 2023-07-02T09:01:21.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.340+0000 7fad72e8e700 1 -- start start 2023-07-02T09:01:21.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.340+0000 7fad72e8e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c132f50 0x7fad6c12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:21.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.340+0000 7fad72e8e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c06d260 con 0x7fad6c132f50 2023-07-02T09:01:21.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad70c2a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c132f50 0x7fad6c12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:21.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad70c2a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c132f50 0x7fad6c12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41216/0 (socket says 172.21.15.103:41216) 2023-07-02T09:01:21.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad70c2a700 1 -- 172.21.15.103:0/2768244927 learned_addr learned my addr 172.21.15.103:0/2768244927 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:21.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad70c2a700 1 -- 172.21.15.103:0/2768244927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c12bb90 con 0x7fad6c132f50 2023-07-02T09:01:21.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad70c2a700 1 --2- 172.21.15.103:0/2768244927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c132f50 0x7fad6c12bea0 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:21.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad697fa700 1 -- 172.21.15.103:0/2768244927 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fad60005a60 con 0x7fad6c132f50 2023-07-02T09:01:21.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad697fa700 1 -- 172.21.15.103:0/2768244927 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fad60005bc0 con 0x7fad6c132f50 2023-07-02T09:01:21.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad697fa700 1 -- 172.21.15.103:0/2768244927 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fad60005e60 con 0x7fad6c132f50 2023-07-02T09:01:21.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6c06f730 con 0x7fad6c132f50 2023-07-02T09:01:21.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.341+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6c06f490 con 0x7fad6c132f50 2023-07-02T09:01:21.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.342+0000 7fad697fa700 1 -- 172.21.15.103:0/2768244927 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fad600070b0 con 0x7fad6c132f50 2023-07-02T09:01:21.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.342+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6c061ea0 con 0x7fad6c132f50 2023-07-02T09:01:21.346 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.342+0000 7fad697fa700 1 --2- 172.21.15.103:0/2768244927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fad500377c0 0x7fad50039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:21.346 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.342+0000 7fad697fa700 1 -- 172.21.15.103:0/2768244927 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fad600469f0 con 0x7fad6c132f50 2023-07-02T09:01:21.346 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.343+0000 7fad6bfff700 1 --2- 172.21.15.103:0/2768244927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fad500377c0 0x7fad50039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:21.346 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.343+0000 7fad6bfff700 1 --2- 172.21.15.103:0/2768244927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fad500377c0 0x7fad50039cd0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:21.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.347+0000 7fad697fa700 1 -- 172.21.15.103:0/2768244927 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fad6000a4a0 con 0x7fad6c132f50 2023-07-02T09:01:21.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.535+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fad6c06f490 con 0x7fad6c132f50 2023-07-02T09:01:21.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.535+0000 7fad697fa700 1 -- 172.21.15.103:0/2768244927 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fad6c06f490 con 0x7fad6c132f50 2023-07-02T09:01:21.536 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:21.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fad500377c0 msgr2=0x7fad50039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 --2- 172.21.15.103:0/2768244927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fad500377c0 0x7fad50039cd0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c132f50 msgr2=0x7fad6c12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 --2- 172.21.15.103:0/2768244927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c132f50 0x7fad6c12bea0 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 shutdown_connections 2023-07-02T09:01:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 --2- 172.21.15.103:0/2768244927 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fad500377c0 0x7fad50039cd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 --2- 172.21.15.103:0/2768244927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad6c132f50 0x7fad6c12bea0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:21.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 >> 172.21.15.103:0/2768244927 conn(0x7fad6c0a2080 msgr2=0x7fad6c1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:21.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 shutdown_connections 2023-07-02T09:01:21.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:21.537+0000 7fad72e8e700 1 -- 172.21.15.103:0/2768244927 wait complete. 2023-07-02T09:01:21.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:22.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:22.695 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.691+0000 7f8b19ae5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:22.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.696+0000 7f8b19ae5700 1 Processor -- start 2023-07-02T09:01:22.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.696+0000 7f8b19ae5700 1 -- start start 2023-07-02T09:01:22.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b19ae5700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8b14132f50 0x7f8b14133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:22.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b19ae5700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1409e260 con 0x7f8b1409c580 2023-07-02T09:01:22.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b19ae5700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b140a1640 con 0x7f8b14132f50 2023-07-02T09:01:22.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b137fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8b14132f50 0x7f8b14133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:22.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b12ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f8b1409c580 0x7f8b1409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40672/0 (socket says 172.21.15.103:40672) 2023-07-02T09:01:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b12ffd700 1 -- 172.21.15.103:0/844206350 learned_addr learned my addr 172.21.15.103:0/844206350 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b137fe700 1 -- 172.21.15.103:0/844206350 >> v1:172.21.15.103:6789/0 conn(0x7f8b1409c580 legacy=0x7f8b1409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b137fe700 1 -- 172.21.15.103:0/844206350 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b14132a20 con 0x7f8b14132f50 2023-07-02T09:01:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b127fc700 1 -- 172.21.15.103:0/844206350 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294544549 0 0) 0x7f8b1409e260 con 0x7f8b1409c580 2023-07-02T09:01:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.697+0000 7f8b137fe700 1 --2- 172.21.15.103:0/844206350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b14132f50 0x7f8b14133360 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ae3ca06b6768b4 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b127fc700 1 -- 172.21.15.103:0/844206350 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8b08009cd0 con 0x7f8b14132f50 2023-07-02T09:01:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b127fc700 1 -- 172.21.15.103:0/844206350 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8b08009e30 con 0x7f8b14132f50 2023-07-02T09:01:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b127fc700 1 -- 172.21.15.103:0/844206350 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8b0800e360 con 0x7f8b14132f50 2023-07-02T09:01:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b19ae5700 1 -- 172.21.15.103:0/844206350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b14132f50 msgr2=0x7f8b14133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b19ae5700 1 --2- 172.21.15.103:0/844206350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b14132f50 0x7f8b14133360 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b19ae5700 1 -- 172.21.15.103:0/844206350 shutdown_connections 2023-07-02T09:01:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b19ae5700 1 --2- 172.21.15.103:0/844206350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b14132f50 0x7f8b14133360 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b19ae5700 1 -- 172.21.15.103:0/844206350 >> 172.21.15.103:0/844206350 conn(0x7f8b140a2080 msgr2=0x7f8b140a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b19ae5700 1 -- 172.21.15.103:0/844206350 shutdown_connections 2023-07-02T09:01:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.698+0000 7f8b19ae5700 1 -- 172.21.15.103:0/844206350 wait complete. 2023-07-02T09:01:22.702 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.699+0000 7f8b19ae5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:22.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.699+0000 7f8b19ae5700 1 Processor -- start 2023-07-02T09:01:22.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.700+0000 7f8b19ae5700 1 -- start start 2023-07-02T09:01:22.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.700+0000 7f8b19ae5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1409c580 0x7f8b141baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:22.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.700+0000 7f8b19ae5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b141ba610 con 0x7f8b1409c580 2023-07-02T09:01:22.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.700+0000 7f8b137fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1409c580 0x7f8b141baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:22.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8b137fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1409c580 0x7f8b141baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41222/0 (socket says 172.21.15.103:41222) 2023-07-02T09:01:22.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8b137fe700 1 -- 172.21.15.103:0/356353788 learned_addr learned my addr 172.21.15.103:0/356353788 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:22.704 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8b137fe700 1 -- 172.21.15.103:0/356353788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1406dfc0 con 0x7f8b1409c580 2023-07-02T09:01:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8b137fe700 1 --2- 172.21.15.103:0/356353788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1409c580 0x7f8b141baa60 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8afbfff700 1 -- 172.21.15.103:0/356353788 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8b08013530 con 0x7f8b1409c580 2023-07-02T09:01:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8afbfff700 1 -- 172.21.15.103:0/356353788 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8b08013690 con 0x7f8b1409c580 2023-07-02T09:01:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8afbfff700 1 -- 172.21.15.103:0/356353788 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8b08013930 con 0x7f8b1409c580 2023-07-02T09:01:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b141ba270 con 0x7f8b1409c580 2023-07-02T09:01:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.701+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1406dba0 con 0x7f8b1409c580 2023-07-02T09:01:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.702+0000 7f8afbfff700 1 -- 172.21.15.103:0/356353788 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f8b08013ae0 con 0x7f8b1409c580 2023-07-02T09:01:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.702+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b141ba460 con 0x7f8b1409c580 2023-07-02T09:01:22.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.702+0000 7f8afbfff700 1 --2- 172.21.15.103:0/356353788 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b000377b0 0x7f8b00039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:22.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.702+0000 7f8afbfff700 1 -- 172.21.15.103:0/356353788 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f8b08018040 con 0x7f8b1409c580 2023-07-02T09:01:22.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.703+0000 7f8b12ffd700 1 --2- 172.21.15.103:0/356353788 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b000377b0 0x7f8b00039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:22.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.703+0000 7f8b12ffd700 1 --2- 172.21.15.103:0/356353788 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b000377b0 0x7f8b00039cc0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:22.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.707+0000 7f8afbfff700 1 -- 172.21.15.103:0/356353788 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f8b08018e00 con 0x7f8b1409c580 2023-07-02T09:01:22.901 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.901+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8b14061ea0 con 0x7f8b1409c580 2023-07-02T09:01:22.902 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.901+0000 7f8afbfff700 1 -- 172.21.15.103:0/356353788 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f8b08010e60 con 0x7f8b1409c580 2023-07-02T09:01:22.902 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b000377b0 msgr2=0x7f8b00039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 --2- 172.21.15.103:0/356353788 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b000377b0 0x7f8b00039cc0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1409c580 msgr2=0x7f8b141baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 --2- 172.21.15.103:0/356353788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1409c580 0x7f8b141baa60 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 shutdown_connections 2023-07-02T09:01:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 --2- 172.21.15.103:0/356353788 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8b000377b0 0x7f8b00039cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:22.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 --2- 172.21.15.103:0/356353788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1409c580 0x7f8b141baa60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:22.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 >> 172.21.15.103:0/356353788 conn(0x7f8b140a2080 msgr2=0x7f8b14130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:22.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 shutdown_connections 2023-07-02T09:01:22.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:22.903+0000 7f8b19ae5700 1 -- 172.21.15.103:0/356353788 wait complete. 2023-07-02T09:01:22.916 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:23.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:24.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.048+0000 7f314a485700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:24.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.053+0000 7f314a485700 1 Processor -- start 2023-07-02T09:01:24.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.053+0000 7f314a485700 1 -- start start 2023-07-02T09:01:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f314a485700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f314412f250 0x7f314412f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f314a485700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314406df60 con 0x7f3144098a00 2023-07-02T09:01:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f314a485700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f314409a710 con 0x7f314412f250 2023-07-02T09:01:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f3143fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f314412f250 0x7f314412f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f31437fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3144098a00 0x7f3144098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:53832/0 (socket says 172.21.15.103:53832) 2023-07-02T09:01:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f3143fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f314412f250 0x7f314412f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38050/0 (socket says 172.21.15.103:38050) 2023-07-02T09:01:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f31437fe700 1 -- 172.21.15.103:0/1225685665 learned_addr learned my addr 172.21.15.103:0/1225685665 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f3143fff700 1 -- 172.21.15.103:0/1225685665 >> v1:172.21.15.103:6789/0 conn(0x7f3144098a00 legacy=0x7f3144098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f3142ffd700 1 -- 172.21.15.103:0/1225685665 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500095247 0 0) 0x7f314406df60 con 0x7f3144098a00 2023-07-02T09:01:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.054+0000 7f3143fff700 1 -- 172.21.15.103:0/1225685665 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314412ed20 con 0x7f314412f250 2023-07-02T09:01:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f3143fff700 1 --2- 172.21.15.103:0/1225685665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314412f250 0x7f314412f660 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58d3dc28171cd43 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f3142ffd700 1 -- 172.21.15.103:0/1225685665 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f313800ac60 con 0x7f314412f250 2023-07-02T09:01:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f3142ffd700 1 -- 172.21.15.103:0/1225685665 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f313800adc0 con 0x7f314412f250 2023-07-02T09:01:24.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f3142ffd700 1 -- 172.21.15.103:0/1225685665 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f313800f470 con 0x7f314412f250 2023-07-02T09:01:24.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f314a485700 1 -- 172.21.15.103:0/1225685665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314412f250 msgr2=0x7f314412f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:24.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f314a485700 1 --2- 172.21.15.103:0/1225685665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314412f250 0x7f314412f660 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f314a485700 1 -- 172.21.15.103:0/1225685665 shutdown_connections 2023-07-02T09:01:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f314a485700 1 --2- 172.21.15.103:0/1225685665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314412f250 0x7f314412f660 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f314a485700 1 -- 172.21.15.103:0/1225685665 >> 172.21.15.103:0/1225685665 conn(0x7f314409d810 msgr2=0x7f314409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f314a485700 1 -- 172.21.15.103:0/1225685665 shutdown_connections 2023-07-02T09:01:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.055+0000 7f314a485700 1 -- 172.21.15.103:0/1225685665 wait complete. 2023-07-02T09:01:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.056+0000 7f314a485700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.057+0000 7f314a485700 1 Processor -- start 2023-07-02T09:01:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.057+0000 7f314a485700 1 -- start start 2023-07-02T09:01:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.057+0000 7f314a485700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3144098a00 0x7f3144117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.057+0000 7f314a485700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314411a390 con 0x7f3144098a00 2023-07-02T09:01:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f3143fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3144098a00 0x7f3144117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f3143fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3144098a00 0x7f3144117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38062/0 (socket says 172.21.15.103:38062) 2023-07-02T09:01:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f3143fff700 1 -- 172.21.15.103:0/1755155358 learned_addr learned my addr 172.21.15.103:0/1755155358 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f3143fff700 1 -- 172.21.15.103:0/1755155358 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3144099b60 con 0x7f3144098a00 2023-07-02T09:01:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f3143fff700 1 --2- 172.21.15.103:0/1755155358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3144098a00 0x7f3144117360 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:24.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f3140ff9700 1 -- 172.21.15.103:0/1755155358 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3138017040 con 0x7f3144098a00 2023-07-02T09:01:24.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f3140ff9700 1 -- 172.21.15.103:0/1755155358 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f313800fe50 con 0x7f3144098a00 2023-07-02T09:01:24.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f3140ff9700 1 -- 172.21.15.103:0/1755155358 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3138011550 con 0x7f3144098a00 2023-07-02T09:01:24.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31441196f0 con 0x7f3144098a00 2023-07-02T09:01:24.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.058+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3144119db0 con 0x7f3144098a00 2023-07-02T09:01:24.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.059+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3144061ea0 con 0x7f3144098a00 2023-07-02T09:01:24.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.059+0000 7f3140ff9700 1 -- 172.21.15.103:0/1755155358 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3138011700 con 0x7f3144098a00 2023-07-02T09:01:24.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.059+0000 7f3140ff9700 1 --2- 172.21.15.103:0/1755155358 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31300377c0 0x7f3130039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:24.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.059+0000 7f3140ff9700 1 -- 172.21.15.103:0/1755155358 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3138048b40 con 0x7f3144098a00 2023-07-02T09:01:24.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.060+0000 7f31437fe700 1 --2- 172.21.15.103:0/1755155358 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31300377c0 0x7f3130039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:24.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.060+0000 7f31437fe700 1 --2- 172.21.15.103:0/1755155358 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31300377c0 0x7f3130039cd0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:24.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.064+0000 7f3140ff9700 1 -- 172.21.15.103:0/1755155358 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3138018a00 con 0x7f3144098a00 2023-07-02T09:01:24.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.253+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3144099b60 con 0x7f3144098a00 2023-07-02T09:01:24.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.254+0000 7f3140ff9700 1 -- 172.21.15.103:0/1755155358 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3144099b60 con 0x7f3144098a00 2023-07-02T09:01:24.254 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:24.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31300377c0 msgr2=0x7f3130039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 --2- 172.21.15.103:0/1755155358 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31300377c0 0x7f3130039cd0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3144098a00 msgr2=0x7f3144117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 --2- 172.21.15.103:0/1755155358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3144098a00 0x7f3144117360 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 shutdown_connections 2023-07-02T09:01:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 --2- 172.21.15.103:0/1755155358 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31300377c0 0x7f3130039cd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 --2- 172.21.15.103:0/1755155358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3144098a00 0x7f3144117360 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 >> 172.21.15.103:0/1755155358 conn(0x7f314409d810 msgr2=0x7f31440a0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:24.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 shutdown_connections 2023-07-02T09:01:24.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:24.256+0000 7f314a485700 1 -- 172.21.15.103:0/1755155358 wait complete. 2023-07-02T09:01:24.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:24.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~99s 2023-07-02T09:01:25.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:25.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.404+0000 7f76a7b0c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:25.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.409+0000 7f76a7b0c700 1 Processor -- start 2023-07-02T09:01:25.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.409+0000 7f76a7b0c700 1 -- start start 2023-07-02T09:01:25.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a7b0c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f76a0132030 0x7f76a0134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:25.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a7b0c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a006df60 con 0x7f76a012f250 2023-07-02T09:01:25.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a7b0c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a009a710 con 0x7f76a0132030 2023-07-02T09:01:25.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a50a7700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f76a0132030 0x7f76a0134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:25.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a58a8700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f76a012f250 0x7f76a012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:53844/0 (socket says 172.21.15.103:53844) 2023-07-02T09:01:25.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a58a8700 1 -- 172.21.15.103:0/1474850927 learned_addr learned my addr 172.21.15.103:0/1474850927 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:25.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a50a7700 1 -- 172.21.15.103:0/1474850927 >> v1:172.21.15.103:6789/0 conn(0x7f76a012f250 legacy=0x7f76a012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:25.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a50a7700 1 -- 172.21.15.103:0/1474850927 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a012ed20 con 0x7f76a0132030 2023-07-02T09:01:25.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a48a6700 1 -- 172.21.15.103:0/1474850927 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822143328 0 0) 0x7f76a006df60 con 0x7f76a012f250 2023-07-02T09:01:25.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a50a7700 1 --2- 172.21.15.103:0/1474850927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a0132030 0x7f76a0134450 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c0a94d805756b5 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:25.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a48a6700 1 -- 172.21.15.103:0/1474850927 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7690009cd0 con 0x7f76a0132030 2023-07-02T09:01:25.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a48a6700 1 -- 172.21.15.103:0/1474850927 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7690009e30 con 0x7f76a0132030 2023-07-02T09:01:25.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.410+0000 7f76a48a6700 1 -- 172.21.15.103:0/1474850927 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f769000e360 con 0x7f76a0132030 2023-07-02T09:01:25.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.411+0000 7f76a7b0c700 1 -- 172.21.15.103:0/1474850927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a0132030 msgr2=0x7f76a0134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:25.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.411+0000 7f76a7b0c700 1 --2- 172.21.15.103:0/1474850927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a0132030 0x7f76a0134450 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:25.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.411+0000 7f76a7b0c700 1 -- 172.21.15.103:0/1474850927 shutdown_connections 2023-07-02T09:01:25.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.411+0000 7f76a7b0c700 1 --2- 172.21.15.103:0/1474850927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a0132030 0x7f76a0134450 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:25.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.411+0000 7f76a7b0c700 1 -- 172.21.15.103:0/1474850927 >> 172.21.15.103:0/1474850927 conn(0x7f76a009d810 msgr2=0x7f76a009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:25.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.411+0000 7f76a7b0c700 1 -- 172.21.15.103:0/1474850927 shutdown_connections 2023-07-02T09:01:25.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.411+0000 7f76a7b0c700 1 -- 172.21.15.103:0/1474850927 wait complete. 2023-07-02T09:01:25.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.411+0000 7f76a7b0c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:25.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.412+0000 7f76a7b0c700 1 Processor -- start 2023-07-02T09:01:25.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.412+0000 7f76a7b0c700 1 -- start start 2023-07-02T09:01:25.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.413+0000 7f76a7b0c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a012f250 0x7f76a00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:25.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.413+0000 7f76a7b0c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a0118c20 con 0x7f76a012f250 2023-07-02T09:01:25.415 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.413+0000 7f76a58a8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a012f250 0x7f76a00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:25.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.413+0000 7f76a58a8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a012f250 0x7f76a00705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38076/0 (socket says 172.21.15.103:38076) 2023-07-02T09:01:25.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.413+0000 7f76a58a8700 1 -- 172.21.15.103:0/2951217602 learned_addr learned my addr 172.21.15.103:0/2951217602 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:25.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.413+0000 7f76a58a8700 1 -- 172.21.15.103:0/2951217602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a01196a0 con 0x7f76a012f250 2023-07-02T09:01:25.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.413+0000 7f76a58a8700 1 --2- 172.21.15.103:0/2951217602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a012f250 0x7f76a00705a0 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:25.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.413+0000 7f76967fc700 1 -- 172.21.15.103:0/2951217602 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f769c00ebe0 con 0x7f76a012f250 2023-07-02T09:01:25.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.414+0000 7f76967fc700 1 -- 172.21.15.103:0/2951217602 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f769c00edb0 con 0x7f76a012f250 2023-07-02T09:01:25.416 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.414+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a0072200 con 0x7f76a012f250 2023-07-02T09:01:25.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.414+0000 7f76967fc700 1 -- 172.21.15.103:0/2951217602 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f769c0071f0 con 0x7f76a012f250 2023-07-02T09:01:25.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.414+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a0119240 con 0x7f76a012f250 2023-07-02T09:01:25.417 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.414+0000 7f76967fc700 1 -- 172.21.15.103:0/2951217602 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f769c007350 con 0x7f76a012f250 2023-07-02T09:01:25.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.414+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a0118a40 con 0x7f76a012f250 2023-07-02T09:01:25.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.415+0000 7f76967fc700 1 --2- 172.21.15.103:0/2951217602 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7684037810 0x7f7684039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:25.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.415+0000 7f76967fc700 1 -- 172.21.15.103:0/2951217602 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f769c049740 con 0x7f76a012f250 2023-07-02T09:01:25.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.415+0000 7f76a50a7700 1 --2- 172.21.15.103:0/2951217602 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7684037810 0x7f7684039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:25.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.415+0000 7f76a50a7700 1 --2- 172.21.15.103:0/2951217602 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7684037810 0x7f7684039d20 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:25.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.419+0000 7f76967fc700 1 -- 172.21.15.103:0/2951217602 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f769c0014d0 con 0x7f76a012f250 2023-07-02T09:01:25.609 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.609+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f76a0061ea0 con 0x7f76a012f250 2023-07-02T09:01:25.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.609+0000 7f76967fc700 1 -- 172.21.15.103:0/2951217602 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f769c0010e0 con 0x7f76a012f250 2023-07-02T09:01:25.610 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:25.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.611+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7684037810 msgr2=0x7f7684039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:25.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.611+0000 7f76a7b0c700 1 --2- 172.21.15.103:0/2951217602 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7684037810 0x7f7684039d20 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:25.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.611+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a012f250 msgr2=0x7f76a00705a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:25.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.611+0000 7f76a7b0c700 1 --2- 172.21.15.103:0/2951217602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a012f250 0x7f76a00705a0 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:25.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.612+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 shutdown_connections 2023-07-02T09:01:25.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.612+0000 7f76a7b0c700 1 --2- 172.21.15.103:0/2951217602 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7684037810 0x7f7684039d20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:25.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.612+0000 7f76a7b0c700 1 --2- 172.21.15.103:0/2951217602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76a012f250 0x7f76a00705a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:25.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.612+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 >> 172.21.15.103:0/2951217602 conn(0x7f76a009d810 msgr2=0x7f76a012d660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:25.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.612+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 shutdown_connections 2023-07-02T09:01:25.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:25.612+0000 7f76a7b0c700 1 -- 172.21.15.103:0/2951217602 wait complete. 2023-07-02T09:01:25.625 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:26.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:26.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.765+0000 7f97ce7d5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:26.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.770+0000 7f97ce7d5700 1 Processor -- start 2023-07-02T09:01:26.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.770+0000 7f97ce7d5700 1 -- start start 2023-07-02T09:01:26.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97ce7d5700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97c8132f50 0x7f97c8133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:26.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97ce7d5700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c809e260 con 0x7f97c809c580 2023-07-02T09:01:26.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97ce7d5700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c80a1640 con 0x7f97c8132f50 2023-07-02T09:01:26.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97c8132f50 0x7f97c8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:26.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c77fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f97c809c580 0x7f97c809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:53854/0 (socket says 172.21.15.103:53854) 2023-07-02T09:01:26.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97c8132f50 0x7f97c8133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38086/0 (socket says 172.21.15.103:38086) 2023-07-02T09:01:26.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c77fe700 1 -- 172.21.15.103:0/1680825272 learned_addr learned my addr 172.21.15.103:0/1680825272 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:26.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c7fff700 1 -- 172.21.15.103:0/1680825272 >> v1:172.21.15.103:6789/0 conn(0x7f97c809c580 legacy=0x7f97c809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:26.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c7fff700 1 -- 172.21.15.103:0/1680825272 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c8132a20 con 0x7f97c8132f50 2023-07-02T09:01:26.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c6ffd700 1 -- 172.21.15.103:0/1680825272 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060613073 0 0) 0x7f97c809e260 con 0x7f97c809c580 2023-07-02T09:01:26.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c7fff700 1 --2- 172.21.15.103:0/1680825272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c8132f50 0x7f97c8133360 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598294b631d9c2ab server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:26.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.771+0000 7f97c6ffd700 1 -- 172.21.15.103:0/1680825272 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f97bc009cd0 con 0x7f97c8132f50 2023-07-02T09:01:26.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97c6ffd700 1 -- 172.21.15.103:0/1680825272 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f97bc009e30 con 0x7f97c8132f50 2023-07-02T09:01:26.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97c6ffd700 1 -- 172.21.15.103:0/1680825272 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f97bc00e360 con 0x7f97c8132f50 2023-07-02T09:01:26.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1680825272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c8132f50 msgr2=0x7f97c8133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:26.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97ce7d5700 1 --2- 172.21.15.103:0/1680825272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c8132f50 0x7f97c8133360 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:26.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1680825272 shutdown_connections 2023-07-02T09:01:26.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97ce7d5700 1 --2- 172.21.15.103:0/1680825272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c8132f50 0x7f97c8133360 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:26.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1680825272 >> 172.21.15.103:0/1680825272 conn(0x7f97c80a2080 msgr2=0x7f97c80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:26.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1680825272 shutdown_connections 2023-07-02T09:01:26.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1680825272 wait complete. 2023-07-02T09:01:26.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.772+0000 7f97ce7d5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:26.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.773+0000 7f97ce7d5700 1 Processor -- start 2023-07-02T09:01:26.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.774+0000 7f97ce7d5700 1 -- start start 2023-07-02T09:01:26.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.774+0000 7f97ce7d5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c809c580 0x7f97c81baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:26.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.774+0000 7f97ce7d5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c81ba610 con 0x7f97c809c580 2023-07-02T09:01:26.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.774+0000 7f97c7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c809c580 0x7f97c81baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:26.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.774+0000 7f97c7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c809c580 0x7f97c81baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38094/0 (socket says 172.21.15.103:38094) 2023-07-02T09:01:26.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.774+0000 7f97c7fff700 1 -- 172.21.15.103:0/1914386791 learned_addr learned my addr 172.21.15.103:0/1914386791 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:26.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.775+0000 7f97c7fff700 1 -- 172.21.15.103:0/1914386791 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c806dfc0 con 0x7f97c809c580 2023-07-02T09:01:26.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.775+0000 7f97c7fff700 1 --2- 172.21.15.103:0/1914386791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c809c580 0x7f97c81baa60 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:26.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.775+0000 7f97c4ff9700 1 -- 172.21.15.103:0/1914386791 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f97bc013530 con 0x7f97c809c580 2023-07-02T09:01:26.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.775+0000 7f97c4ff9700 1 -- 172.21.15.103:0/1914386791 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f97bc013690 con 0x7f97c809c580 2023-07-02T09:01:26.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.775+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c81ba270 con 0x7f97c809c580 2023-07-02T09:01:26.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.775+0000 7f97c4ff9700 1 -- 172.21.15.103:0/1914386791 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f97bc013930 con 0x7f97c809c580 2023-07-02T09:01:26.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.775+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c806dba0 con 0x7f97c809c580 2023-07-02T09:01:26.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.776+0000 7f97c4ff9700 1 -- 172.21.15.103:0/1914386791 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f97bc013ae0 con 0x7f97c809c580 2023-07-02T09:01:26.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.776+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c81ba460 con 0x7f97c809c580 2023-07-02T09:01:26.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.776+0000 7f97c4ff9700 1 --2- 172.21.15.103:0/1914386791 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97b40377b0 0x7f97b4039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:26.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.776+0000 7f97c4ff9700 1 -- 172.21.15.103:0/1914386791 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f97bc018040 con 0x7f97c809c580 2023-07-02T09:01:26.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.776+0000 7f97c77fe700 1 --2- 172.21.15.103:0/1914386791 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97b40377b0 0x7f97b4039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:26.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.777+0000 7f97c77fe700 1 --2- 172.21.15.103:0/1914386791 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97b40377b0 0x7f97b4039cc0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:26.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.781+0000 7f97c4ff9700 1 -- 172.21.15.103:0/1914386791 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f97bc018e00 con 0x7f97c809c580 2023-07-02T09:01:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.968+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f97c8061ea0 con 0x7f97c809c580 2023-07-02T09:01:26.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.968+0000 7f97c4ff9700 1 -- 172.21.15.103:0/1914386791 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f97bc010e60 con 0x7f97c809c580 2023-07-02T09:01:26.970 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:26.971 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97b40377b0 msgr2=0x7f97b4039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:26.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 --2- 172.21.15.103:0/1914386791 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97b40377b0 0x7f97b4039cc0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:26.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c809c580 msgr2=0x7f97c81baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:26.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 --2- 172.21.15.103:0/1914386791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c809c580 0x7f97c81baa60 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:26.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 shutdown_connections 2023-07-02T09:01:26.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 --2- 172.21.15.103:0/1914386791 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97b40377b0 0x7f97b4039cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:26.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 --2- 172.21.15.103:0/1914386791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97c809c580 0x7f97c81baa60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:26.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 >> 172.21.15.103:0/1914386791 conn(0x7f97c80a2080 msgr2=0x7f97c8130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:26.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 shutdown_connections 2023-07-02T09:01:26.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:26.970+0000 7f97ce7d5700 1 -- 172.21.15.103:0/1914386791 wait complete. 2023-07-02T09:01:26.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:27.985 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:28.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.124+0000 7f2864031700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:28.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.129+0000 7f2864031700 1 Processor -- start 2023-07-02T09:01:28.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.129+0000 7f2864031700 1 -- start start 2023-07-02T09:01:28.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2864031700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f285c132f50 0x7f285c133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:28.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2864031700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285c09e260 con 0x7f285c09c580 2023-07-02T09:01:28.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2864031700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285c0a1640 con 0x7f285c132f50 2023-07-02T09:01:28.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2861dcd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f285c132f50 0x7f285c133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:28.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f28615cc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f285c09c580 0x7f285c09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:53870/0 (socket says 172.21.15.103:53870) 2023-07-02T09:01:28.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f28615cc700 1 -- 172.21.15.103:0/989910589 learned_addr learned my addr 172.21.15.103:0/989910589 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:28.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2861dcd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f285c132f50 0x7f285c133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38106/0 (socket says 172.21.15.103:38106) 2023-07-02T09:01:28.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2861dcd700 1 -- 172.21.15.103:0/989910589 >> v1:172.21.15.103:6789/0 conn(0x7f285c09c580 legacy=0x7f285c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:28.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2861dcd700 1 -- 172.21.15.103:0/989910589 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c132a20 con 0x7f285c132f50 2023-07-02T09:01:28.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2860dcb700 1 -- 172.21.15.103:0/989910589 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024329155 0 0) 0x7f285c09e260 con 0x7f285c09c580 2023-07-02T09:01:28.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2861dcd700 1 --2- 172.21.15.103:0/989910589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c132f50 0x7f285c133360 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c577a107cbfebdb5 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:28.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2860dcb700 1 -- 172.21.15.103:0/989910589 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2858009cd0 con 0x7f285c132f50 2023-07-02T09:01:28.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2860dcb700 1 -- 172.21.15.103:0/989910589 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2858009e30 con 0x7f285c132f50 2023-07-02T09:01:28.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.130+0000 7f2860dcb700 1 -- 172.21.15.103:0/989910589 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f285800e360 con 0x7f285c132f50 2023-07-02T09:01:28.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.131+0000 7f2864031700 1 -- 172.21.15.103:0/989910589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c132f50 msgr2=0x7f285c133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:28.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.131+0000 7f2864031700 1 --2- 172.21.15.103:0/989910589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c132f50 0x7f285c133360 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:28.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.131+0000 7f2864031700 1 -- 172.21.15.103:0/989910589 shutdown_connections 2023-07-02T09:01:28.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.131+0000 7f2864031700 1 --2- 172.21.15.103:0/989910589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c132f50 0x7f285c133360 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:28.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.131+0000 7f2864031700 1 -- 172.21.15.103:0/989910589 >> 172.21.15.103:0/989910589 conn(0x7f285c0a2080 msgr2=0x7f285c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:28.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.131+0000 7f2864031700 1 -- 172.21.15.103:0/989910589 shutdown_connections 2023-07-02T09:01:28.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.131+0000 7f2864031700 1 -- 172.21.15.103:0/989910589 wait complete. 2023-07-02T09:01:28.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.132+0000 7f2864031700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:28.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.133+0000 7f2864031700 1 Processor -- start 2023-07-02T09:01:28.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.133+0000 7f2864031700 1 -- start start 2023-07-02T09:01:28.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2864031700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c09c580 0x7f285c12beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:28.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2864031700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c133a00 con 0x7f285c09c580 2023-07-02T09:01:28.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2861dcd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c09c580 0x7f285c12beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:28.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2861dcd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c09c580 0x7f285c12beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38112/0 (socket says 172.21.15.103:38112) 2023-07-02T09:01:28.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2861dcd700 1 -- 172.21.15.103:0/1552095798 learned_addr learned my addr 172.21.15.103:0/1552095798 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:28.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2861dcd700 1 -- 172.21.15.103:0/1552095798 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c06dc00 con 0x7f285c09c580 2023-07-02T09:01:28.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2861dcd700 1 --2- 172.21.15.103:0/1552095798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c09c580 0x7f285c12beb0 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:28.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f28527fc700 1 -- 172.21.15.103:0/1552095798 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2858014540 con 0x7f285c09c580 2023-07-02T09:01:28.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f28527fc700 1 -- 172.21.15.103:0/1552095798 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f28580146a0 con 0x7f285c09c580 2023-07-02T09:01:28.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c06e920 con 0x7f285c09c580 2023-07-02T09:01:28.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f28527fc700 1 -- 172.21.15.103:0/1552095798 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2858014940 con 0x7f285c09c580 2023-07-02T09:01:28.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.134+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285c071f80 con 0x7f285c09c580 2023-07-02T09:01:28.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.135+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285c061ea0 con 0x7f285c09c580 2023-07-02T09:01:28.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.135+0000 7f28527fc700 1 -- 172.21.15.103:0/1552095798 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2858014af0 con 0x7f285c09c580 2023-07-02T09:01:28.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.135+0000 7f28527fc700 1 --2- 172.21.15.103:0/1552095798 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f28400377c0 0x7f2840039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:28.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.135+0000 7f28527fc700 1 -- 172.21.15.103:0/1552095798 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f28580483c0 con 0x7f285c09c580 2023-07-02T09:01:28.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.136+0000 7f28615cc700 1 --2- 172.21.15.103:0/1552095798 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f28400377c0 0x7f2840039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:28.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.136+0000 7f28615cc700 1 --2- 172.21.15.103:0/1552095798 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f28400377c0 0x7f2840039cd0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:28.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.140+0000 7f28527fc700 1 -- 172.21.15.103:0/1552095798 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f285800fc20 con 0x7f285c09c580 2023-07-02T09:01:28.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.328+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f285c0725b0 con 0x7f285c09c580 2023-07-02T09:01:28.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.328+0000 7f28527fc700 1 -- 172.21.15.103:0/1552095798 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2858011320 con 0x7f285c09c580 2023-07-02T09:01:28.330 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:28.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f28400377c0 msgr2=0x7f2840039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:28.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 --2- 172.21.15.103:0/1552095798 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f28400377c0 0x7f2840039cd0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:28.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c09c580 msgr2=0x7f285c12beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:28.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 --2- 172.21.15.103:0/1552095798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c09c580 0x7f285c12beb0 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:28.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 shutdown_connections 2023-07-02T09:01:28.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 --2- 172.21.15.103:0/1552095798 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f28400377c0 0x7f2840039cd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:28.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 --2- 172.21.15.103:0/1552095798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285c09c580 0x7f285c12beb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:28.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 >> 172.21.15.103:0/1552095798 conn(0x7f285c0a2080 msgr2=0x7f285c1ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:28.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 shutdown_connections 2023-07-02T09:01:28.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:28.330+0000 7f2864031700 1 -- 172.21.15.103:0/1552095798 wait complete. 2023-07-02T09:01:28.344 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:29.346 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:29.489 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.483+0000 7fc76fd25700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:29.490 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.488+0000 7fc76fd25700 1 Processor -- start 2023-07-02T09:01:29.490 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.489+0000 7fc76fd25700 1 -- start start 2023-07-02T09:01:29.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.489+0000 7fc76fd25700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc768133360 0x7fc76809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:29.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.489+0000 7fc76fd25700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76809e260 con 0x7fc768132f50 2023-07-02T09:01:29.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.489+0000 7fc76fd25700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7680a1640 con 0x7fc768133360 2023-07-02T09:01:29.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.489+0000 7fc76d2c0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc768133360 0x7fc76809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:29.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.489+0000 7fc76dac1700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc768132f50 0x7fc76809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:53876/0 (socket says 172.21.15.103:53876) 2023-07-02T09:01:29.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.489+0000 7fc76dac1700 1 -- 172.21.15.103:0/1442070864 learned_addr learned my addr 172.21.15.103:0/1442070864 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:29.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.490+0000 7fc76d2c0700 1 -- 172.21.15.103:0/1442070864 >> v1:172.21.15.103:6789/0 conn(0x7fc768132f50 legacy=0x7fc76809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:29.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.490+0000 7fc76d2c0700 1 -- 172.21.15.103:0/1442070864 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768132a20 con 0x7fc768133360 2023-07-02T09:01:29.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.490+0000 7fc76cabf700 1 -- 172.21.15.103:0/1442070864 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511876896 0 0) 0x7fc76809e260 con 0x7fc768132f50 2023-07-02T09:01:29.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.490+0000 7fc76d2c0700 1 --2- 172.21.15.103:0/1442070864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768133360 0x7fc76809cc90 crc :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bfce1471e81f70 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:29.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.490+0000 7fc76cabf700 1 -- 172.21.15.103:0/1442070864 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc764009cd0 con 0x7fc768133360 2023-07-02T09:01:29.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.490+0000 7fc76cabf700 1 -- 172.21.15.103:0/1442070864 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc764009e30 con 0x7fc768133360 2023-07-02T09:01:29.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.490+0000 7fc76cabf700 1 -- 172.21.15.103:0/1442070864 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc76400e360 con 0x7fc768133360 2023-07-02T09:01:29.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.491+0000 7fc76fd25700 1 -- 172.21.15.103:0/1442070864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768133360 msgr2=0x7fc76809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:29.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.491+0000 7fc76fd25700 1 --2- 172.21.15.103:0/1442070864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768133360 0x7fc76809cc90 crc :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:29.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.491+0000 7fc76fd25700 1 -- 172.21.15.103:0/1442070864 shutdown_connections 2023-07-02T09:01:29.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.491+0000 7fc76fd25700 1 --2- 172.21.15.103:0/1442070864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768133360 0x7fc76809cc90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:29.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.491+0000 7fc76fd25700 1 -- 172.21.15.103:0/1442070864 >> 172.21.15.103:0/1442070864 conn(0x7fc7680a2080 msgr2=0x7fc7680a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:29.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.491+0000 7fc76fd25700 1 -- 172.21.15.103:0/1442070864 shutdown_connections 2023-07-02T09:01:29.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.491+0000 7fc76fd25700 1 -- 172.21.15.103:0/1442070864 wait complete. 2023-07-02T09:01:29.496 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.491+0000 7fc76fd25700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:29.496 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.492+0000 7fc76fd25700 1 Processor -- start 2023-07-02T09:01:29.496 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.492+0000 7fc76fd25700 1 -- start start 2023-07-02T09:01:29.496 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76fd25700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768132f50 0x7fc76812c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:29.496 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76fd25700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76806d2f0 con 0x7fc768132f50 2023-07-02T09:01:29.496 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76dac1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768132f50 0x7fc76812c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:29.496 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76dac1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768132f50 0x7fc76812c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38130/0 (socket says 172.21.15.103:38130) 2023-07-02T09:01:29.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76dac1700 1 -- 172.21.15.103:0/4290502150 learned_addr learned my addr 172.21.15.103:0/4290502150 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:29.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76dac1700 1 -- 172.21.15.103:0/4290502150 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76806dd60 con 0x7fc768132f50 2023-07-02T09:01:29.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76dac1700 1 --2- 172.21.15.103:0/4290502150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768132f50 0x7fc76812c1a0 crc :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:29.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc75e7fc700 1 -- 172.21.15.103:0/4290502150 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc758004b80 con 0x7fc768132f50 2023-07-02T09:01:29.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc75e7fc700 1 -- 172.21.15.103:0/4290502150 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc758004ce0 con 0x7fc768132f50 2023-07-02T09:01:29.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76812bc50 con 0x7fc768132f50 2023-07-02T09:01:29.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc75e7fc700 1 -- 172.21.15.103:0/4290502150 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc75800c4f0 con 0x7fc768132f50 2023-07-02T09:01:29.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.493+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76806f850 con 0x7fc768132f50 2023-07-02T09:01:29.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.494+0000 7fc75e7fc700 1 -- 172.21.15.103:0/4290502150 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fc75800c650 con 0x7fc768132f50 2023-07-02T09:01:29.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.494+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76806dbb0 con 0x7fc768132f50 2023-07-02T09:01:29.499 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.494+0000 7fc75e7fc700 1 --2- 172.21.15.103:0/4290502150 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc74c0377c0 0x7fc74c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:29.499 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.494+0000 7fc75e7fc700 1 -- 172.21.15.103:0/4290502150 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc758046970 con 0x7fc768132f50 2023-07-02T09:01:29.499 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.495+0000 7fc76d2c0700 1 --2- 172.21.15.103:0/4290502150 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc74c0377c0 0x7fc74c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:29.499 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.495+0000 7fc76d2c0700 1 --2- 172.21.15.103:0/4290502150 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc74c0377c0 0x7fc74c039cd0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:29.500 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.499+0000 7fc75e7fc700 1 -- 172.21.15.103:0/4290502150 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc758009c80 con 0x7fc768132f50 2023-07-02T09:01:29.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.686+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc768061ea0 con 0x7fc768132f50 2023-07-02T09:01:29.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.687+0000 7fc75e7fc700 1 -- 172.21.15.103:0/4290502150 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fc7580098a0 con 0x7fc768132f50 2023-07-02T09:01:29.688 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:29.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc74c0377c0 msgr2=0x7fc74c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:29.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 --2- 172.21.15.103:0/4290502150 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc74c0377c0 0x7fc74c039cd0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:29.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768132f50 msgr2=0x7fc76812c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:29.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 --2- 172.21.15.103:0/4290502150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768132f50 0x7fc76812c1a0 crc :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:29.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 shutdown_connections 2023-07-02T09:01:29.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 --2- 172.21.15.103:0/4290502150 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc74c0377c0 0x7fc74c039cd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:29.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 --2- 172.21.15.103:0/4290502150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc768132f50 0x7fc76812c1a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:29.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 >> 172.21.15.103:0/4290502150 conn(0x7fc7680a2080 msgr2=0x7fc768130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:29.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 shutdown_connections 2023-07-02T09:01:29.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:29.689+0000 7fc76fd25700 1 -- 172.21.15.103:0/4290502150 wait complete. 2023-07-02T09:01:29.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:30.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~105s 2023-07-02T09:01:30.704 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:30.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.836+0000 7ff62e6c9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:30.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.841+0000 7ff62e6c9700 1 Processor -- start 2023-07-02T09:01:30.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.841+0000 7ff62e6c9700 1 -- start start 2023-07-02T09:01:30.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.842+0000 7ff62e6c9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff628132f50 0x7ff628133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:30.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.842+0000 7ff62e6c9700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62809e260 con 0x7ff62809c580 2023-07-02T09:01:30.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.842+0000 7ff62e6c9700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6280a1640 con 0x7ff628132f50 2023-07-02T09:01:30.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.842+0000 7ff627fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff628132f50 0x7ff628133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:30.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.842+0000 7ff6277fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff62809c580 0x7ff62809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:53886/0 (socket says 172.21.15.103:53886) 2023-07-02T09:01:30.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.842+0000 7ff627fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff628132f50 0x7ff628133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38144/0 (socket says 172.21.15.103:38144) 2023-07-02T09:01:30.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.842+0000 7ff6277fe700 1 -- 172.21.15.103:0/1896297139 learned_addr learned my addr 172.21.15.103:0/1896297139 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:30.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.842+0000 7ff627fff700 1 -- 172.21.15.103:0/1896297139 >> v1:172.21.15.103:6789/0 conn(0x7ff62809c580 legacy=0x7ff62809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:30.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff627fff700 1 -- 172.21.15.103:0/1896297139 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff628132a20 con 0x7ff628132f50 2023-07-02T09:01:30.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff626ffd700 1 -- 172.21.15.103:0/1896297139 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020164085 0 0) 0x7ff62809e260 con 0x7ff62809c580 2023-07-02T09:01:30.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff627fff700 1 --2- 172.21.15.103:0/1896297139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff628132f50 0x7ff628133360 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1c595bb5ee5652 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:30.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff626ffd700 1 -- 172.21.15.103:0/1896297139 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff61c009cd0 con 0x7ff628132f50 2023-07-02T09:01:30.846 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff626ffd700 1 -- 172.21.15.103:0/1896297139 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff61c009e30 con 0x7ff628132f50 2023-07-02T09:01:30.846 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff626ffd700 1 -- 172.21.15.103:0/1896297139 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff61c00e360 con 0x7ff628132f50 2023-07-02T09:01:30.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff62e6c9700 1 -- 172.21.15.103:0/1896297139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff628132f50 msgr2=0x7ff628133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:30.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff62e6c9700 1 --2- 172.21.15.103:0/1896297139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff628132f50 0x7ff628133360 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:30.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff62e6c9700 1 -- 172.21.15.103:0/1896297139 shutdown_connections 2023-07-02T09:01:30.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff62e6c9700 1 --2- 172.21.15.103:0/1896297139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff628132f50 0x7ff628133360 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:30.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.843+0000 7ff62e6c9700 1 -- 172.21.15.103:0/1896297139 >> 172.21.15.103:0/1896297139 conn(0x7ff6280a2080 msgr2=0x7ff6280a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:30.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.844+0000 7ff62e6c9700 1 -- 172.21.15.103:0/1896297139 shutdown_connections 2023-07-02T09:01:30.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.844+0000 7ff62e6c9700 1 -- 172.21.15.103:0/1896297139 wait complete. 2023-07-02T09:01:30.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.844+0000 7ff62e6c9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:30.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.845+0000 7ff62e6c9700 1 Processor -- start 2023-07-02T09:01:30.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.845+0000 7ff62e6c9700 1 -- start start 2023-07-02T09:01:30.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.845+0000 7ff62e6c9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62809c580 0x7ff6281baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:30.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.845+0000 7ff62e6c9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6281ba610 con 0x7ff62809c580 2023-07-02T09:01:30.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff627fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62809c580 0x7ff6281baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:30.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff627fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62809c580 0x7ff6281baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38152/0 (socket says 172.21.15.103:38152) 2023-07-02T09:01:30.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff627fff700 1 -- 172.21.15.103:0/2390450383 learned_addr learned my addr 172.21.15.103:0/2390450383 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:30.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff627fff700 1 -- 172.21.15.103:0/2390450383 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62806dfc0 con 0x7ff62809c580 2023-07-02T09:01:30.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff627fff700 1 --2- 172.21.15.103:0/2390450383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62809c580 0x7ff6281baa60 crc :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:30.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff624ff9700 1 -- 172.21.15.103:0/2390450383 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff61c013530 con 0x7ff62809c580 2023-07-02T09:01:30.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff624ff9700 1 -- 172.21.15.103:0/2390450383 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff61c013690 con 0x7ff62809c580 2023-07-02T09:01:30.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6281ba270 con 0x7ff62809c580 2023-07-02T09:01:30.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff624ff9700 1 -- 172.21.15.103:0/2390450383 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff61c013930 con 0x7ff62809c580 2023-07-02T09:01:30.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.846+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62806dba0 con 0x7ff62809c580 2023-07-02T09:01:30.851 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.847+0000 7ff624ff9700 1 -- 172.21.15.103:0/2390450383 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ff61c013ae0 con 0x7ff62809c580 2023-07-02T09:01:30.851 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.847+0000 7ff624ff9700 1 --2- 172.21.15.103:0/2390450383 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff6140377b0 0x7ff614039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:30.851 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.847+0000 7ff624ff9700 1 -- 172.21.15.103:0/2390450383 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ff61c018040 con 0x7ff62809c580 2023-07-02T09:01:30.851 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.847+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6281ba460 con 0x7ff62809c580 2023-07-02T09:01:30.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.847+0000 7ff6277fe700 1 --2- 172.21.15.103:0/2390450383 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff6140377b0 0x7ff614039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:30.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.848+0000 7ff6277fe700 1 --2- 172.21.15.103:0/2390450383 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff6140377b0 0x7ff614039cc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:30.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:30.852+0000 7ff624ff9700 1 -- 172.21.15.103:0/2390450383 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ff61c018e00 con 0x7ff62809c580 2023-07-02T09:01:31.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.043+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff628061ea0 con 0x7ff62809c580 2023-07-02T09:01:31.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.044+0000 7ff624ff9700 1 -- 172.21.15.103:0/2390450383 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ff61c010e60 con 0x7ff62809c580 2023-07-02T09:01:31.045 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:31.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff6140377b0 msgr2=0x7ff614039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:31.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 --2- 172.21.15.103:0/2390450383 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff6140377b0 0x7ff614039cc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:31.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62809c580 msgr2=0x7ff6281baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:31.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 --2- 172.21.15.103:0/2390450383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62809c580 0x7ff6281baa60 crc :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:31.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 shutdown_connections 2023-07-02T09:01:31.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 --2- 172.21.15.103:0/2390450383 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff6140377b0 0x7ff614039cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:31.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 --2- 172.21.15.103:0/2390450383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff62809c580 0x7ff6281baa60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:31.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 >> 172.21.15.103:0/2390450383 conn(0x7ff6280a2080 msgr2=0x7ff628130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:31.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 shutdown_connections 2023-07-02T09:01:31.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:31.046+0000 7ff62e6c9700 1 -- 172.21.15.103:0/2390450383 wait complete. 2023-07-02T09:01:31.060 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:32.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:32.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.198+0000 7f03573f0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:32.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.203+0000 7f03573f0700 1 Processor -- start 2023-07-02T09:01:32.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.203+0000 7f03573f0700 1 -- start start 2023-07-02T09:01:32.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.204+0000 7f03573f0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0350132f50 0x7f0350133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:32.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.204+0000 7f03573f0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035009e260 con 0x7f035009c580 2023-07-02T09:01:32.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.204+0000 7f03573f0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03500a1640 con 0x7f0350132f50 2023-07-02T09:01:32.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.204+0000 7f035518c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0350132f50 0x7f0350133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:32.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.204+0000 7f035498b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f035009c580 0x7f035009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:53890/0 (socket says 172.21.15.103:53890) 2023-07-02T09:01:32.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.204+0000 7f035498b700 1 -- 172.21.15.103:0/2655852159 learned_addr learned my addr 172.21.15.103:0/2655852159 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:32.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.205+0000 7f035518c700 1 -- 172.21.15.103:0/2655852159 >> v1:172.21.15.103:6789/0 conn(0x7f035009c580 legacy=0x7f035009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:32.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.205+0000 7f035518c700 1 -- 172.21.15.103:0/2655852159 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0350132a20 con 0x7f0350132f50 2023-07-02T09:01:32.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.205+0000 7f0347fff700 1 -- 172.21.15.103:0/2655852159 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888353801 0 0) 0x7f035009e260 con 0x7f035009c580 2023-07-02T09:01:32.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.205+0000 7f035518c700 1 --2- 172.21.15.103:0/2655852159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0350132f50 0x7f0350133360 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8adfb978a096a053 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:32.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.205+0000 7f0347fff700 1 -- 172.21.15.103:0/2655852159 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0340009cd0 con 0x7f0350132f50 2023-07-02T09:01:32.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.205+0000 7f0347fff700 1 -- 172.21.15.103:0/2655852159 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0340009e30 con 0x7f0350132f50 2023-07-02T09:01:32.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.205+0000 7f0347fff700 1 -- 172.21.15.103:0/2655852159 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f034000e360 con 0x7f0350132f50 2023-07-02T09:01:32.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.206+0000 7f03573f0700 1 -- 172.21.15.103:0/2655852159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0350132f50 msgr2=0x7f0350133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:32.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.206+0000 7f03573f0700 1 --2- 172.21.15.103:0/2655852159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0350132f50 0x7f0350133360 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:32.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.206+0000 7f03573f0700 1 -- 172.21.15.103:0/2655852159 shutdown_connections 2023-07-02T09:01:32.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.206+0000 7f03573f0700 1 --2- 172.21.15.103:0/2655852159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0350132f50 0x7f0350133360 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:32.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.206+0000 7f03573f0700 1 -- 172.21.15.103:0/2655852159 >> 172.21.15.103:0/2655852159 conn(0x7f03500a2080 msgr2=0x7f03500a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:32.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.206+0000 7f03573f0700 1 -- 172.21.15.103:0/2655852159 shutdown_connections 2023-07-02T09:01:32.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.206+0000 7f03573f0700 1 -- 172.21.15.103:0/2655852159 wait complete. 2023-07-02T09:01:32.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.206+0000 7f03573f0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.207+0000 7f03573f0700 1 Processor -- start 2023-07-02T09:01:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.207+0000 7f03573f0700 1 -- start start 2023-07-02T09:01:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.207+0000 7f03573f0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035009c580 0x7f035012beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.207+0000 7f03573f0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0350133a00 con 0x7f035009c580 2023-07-02T09:01:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f035518c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035009c580 0x7f035012beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f035518c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035009c580 0x7f035012beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38180/0 (socket says 172.21.15.103:38180) 2023-07-02T09:01:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f035518c700 1 -- 172.21.15.103:0/862076260 learned_addr learned my addr 172.21.15.103:0/862076260 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f035518c700 1 -- 172.21.15.103:0/862076260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035006dc00 con 0x7f035009c580 2023-07-02T09:01:32.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f035518c700 1 --2- 172.21.15.103:0/862076260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035009c580 0x7f035012beb0 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:32.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f0345ffb700 1 -- 172.21.15.103:0/862076260 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0340014540 con 0x7f035009c580 2023-07-02T09:01:32.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f0345ffb700 1 -- 172.21.15.103:0/862076260 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f03400146a0 con 0x7f035009c580 2023-07-02T09:01:32.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035006e920 con 0x7f035009c580 2023-07-02T09:01:32.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f0345ffb700 1 -- 172.21.15.103:0/862076260 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0340014940 con 0x7f035009c580 2023-07-02T09:01:32.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.208+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0350071f80 con 0x7f035009c580 2023-07-02T09:01:32.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.209+0000 7f0345ffb700 1 -- 172.21.15.103:0/862076260 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f0340014af0 con 0x7f035009c580 2023-07-02T09:01:32.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.209+0000 7f0345ffb700 1 --2- 172.21.15.103:0/862076260 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03340377c0 0x7f0334039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:32.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.209+0000 7f0345ffb700 1 -- 172.21.15.103:0/862076260 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f0340048340 con 0x7f035009c580 2023-07-02T09:01:32.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.209+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0350061ea0 con 0x7f035009c580 2023-07-02T09:01:32.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.210+0000 7f035498b700 1 --2- 172.21.15.103:0/862076260 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03340377c0 0x7f0334039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:32.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.210+0000 7f035498b700 1 --2- 172.21.15.103:0/862076260 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03340377c0 0x7f0334039cd0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:32.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.214+0000 7f0345ffb700 1 -- 172.21.15.103:0/862076260 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f0340014df0 con 0x7f035009c580 2023-07-02T09:01:32.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.405+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f03500725b0 con 0x7f035009c580 2023-07-02T09:01:32.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.406+0000 7f0345ffb700 1 -- 172.21.15.103:0/862076260 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f0340011320 con 0x7f035009c580 2023-07-02T09:01:32.408 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:32.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03340377c0 msgr2=0x7f0334039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:32.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 --2- 172.21.15.103:0/862076260 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03340377c0 0x7f0334039cd0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:32.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035009c580 msgr2=0x7f035012beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:32.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 --2- 172.21.15.103:0/862076260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035009c580 0x7f035012beb0 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:32.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 shutdown_connections 2023-07-02T09:01:32.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 --2- 172.21.15.103:0/862076260 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03340377c0 0x7f0334039cd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:32.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 --2- 172.21.15.103:0/862076260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035009c580 0x7f035012beb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:32.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 >> 172.21.15.103:0/862076260 conn(0x7f03500a2080 msgr2=0x7f03501ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:32.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 shutdown_connections 2023-07-02T09:01:32.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:32.408+0000 7f03573f0700 1 -- 172.21.15.103:0/862076260 wait complete. 2023-07-02T09:01:32.422 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:33.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:33.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.561+0000 7fb2e4296700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:33.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.566+0000 7fb2e4296700 1 Processor -- start 2023-07-02T09:01:33.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.566+0000 7fb2e4296700 1 -- start start 2023-07-02T09:01:33.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.567+0000 7fb2e4296700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2dc133360 0x7fb2dc09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:33.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.567+0000 7fb2e4296700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc09e260 con 0x7fb2dc132f50 2023-07-02T09:01:33.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.567+0000 7fb2e4296700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc0a1640 con 0x7fb2dc133360 2023-07-02T09:01:33.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.567+0000 7fb2e1831700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2dc133360 0x7fb2dc09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:33.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.567+0000 7fb2e2032700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb2dc132f50 0x7fb2dc09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37072/0 (socket says 172.21.15.103:37072) 2023-07-02T09:01:33.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.567+0000 7fb2e1831700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2dc133360 0x7fb2dc09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43860/0 (socket says 172.21.15.103:43860) 2023-07-02T09:01:33.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.567+0000 7fb2e2032700 1 -- 172.21.15.103:0/3818307788 learned_addr learned my addr 172.21.15.103:0/3818307788 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:33.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.568+0000 7fb2e1831700 1 -- 172.21.15.103:0/3818307788 >> v1:172.21.15.103:6789/0 conn(0x7fb2dc132f50 legacy=0x7fb2dc09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:33.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.568+0000 7fb2e1831700 1 -- 172.21.15.103:0/3818307788 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc132a20 con 0x7fb2dc133360 2023-07-02T09:01:33.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.568+0000 7fb2e1831700 1 --2- 172.21.15.103:0/3818307788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc133360 0x7fb2dc09cc90 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f7192d0152b8f2 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:33.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.568+0000 7fb2e1030700 1 -- 172.21.15.103:0/3818307788 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb2d8009cd0 con 0x7fb2dc133360 2023-07-02T09:01:33.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.568+0000 7fb2e1030700 1 -- 172.21.15.103:0/3818307788 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb2d8009e30 con 0x7fb2dc133360 2023-07-02T09:01:33.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.568+0000 7fb2e1030700 1 -- 172.21.15.103:0/3818307788 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb2d800e360 con 0x7fb2dc133360 2023-07-02T09:01:33.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.569+0000 7fb2e4296700 1 -- 172.21.15.103:0/3818307788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc133360 msgr2=0x7fb2dc09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:33.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.569+0000 7fb2e4296700 1 --2- 172.21.15.103:0/3818307788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc133360 0x7fb2dc09cc90 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:33.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.569+0000 7fb2e4296700 1 -- 172.21.15.103:0/3818307788 shutdown_connections 2023-07-02T09:01:33.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.569+0000 7fb2e4296700 1 --2- 172.21.15.103:0/3818307788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc133360 0x7fb2dc09cc90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:33.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.569+0000 7fb2e4296700 1 -- 172.21.15.103:0/3818307788 >> 172.21.15.103:0/3818307788 conn(0x7fb2dc0a2080 msgr2=0x7fb2dc0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:33.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.569+0000 7fb2e4296700 1 -- 172.21.15.103:0/3818307788 shutdown_connections 2023-07-02T09:01:33.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.569+0000 7fb2e4296700 1 -- 172.21.15.103:0/3818307788 wait complete. 2023-07-02T09:01:33.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.569+0000 7fb2e4296700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:33.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.570+0000 7fb2e4296700 1 Processor -- start 2023-07-02T09:01:33.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.570+0000 7fb2e4296700 1 -- start start 2023-07-02T09:01:33.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.571+0000 7fb2e4296700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc132f50 0x7fb2dc12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:33.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.571+0000 7fb2e4296700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc06d260 con 0x7fb2dc132f50 2023-07-02T09:01:33.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.571+0000 7fb2e2032700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc132f50 0x7fb2dc12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:33.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.571+0000 7fb2e2032700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc132f50 0x7fb2dc12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43872/0 (socket says 172.21.15.103:43872) 2023-07-02T09:01:33.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.571+0000 7fb2e2032700 1 -- 172.21.15.103:0/1159625722 learned_addr learned my addr 172.21.15.103:0/1159625722 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:33.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.572+0000 7fb2e2032700 1 -- 172.21.15.103:0/1159625722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc12bb90 con 0x7fb2dc132f50 2023-07-02T09:01:33.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.572+0000 7fb2e2032700 1 --2- 172.21.15.103:0/1159625722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc132f50 0x7fb2dc12bea0 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:33.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.572+0000 7fb2d2ffd700 1 -- 172.21.15.103:0/1159625722 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb2cc005a60 con 0x7fb2dc132f50 2023-07-02T09:01:33.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.572+0000 7fb2d2ffd700 1 -- 172.21.15.103:0/1159625722 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb2cc005bc0 con 0x7fb2dc132f50 2023-07-02T09:01:33.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.572+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc06f730 con 0x7fb2dc132f50 2023-07-02T09:01:33.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.572+0000 7fb2d2ffd700 1 -- 172.21.15.103:0/1159625722 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb2cc005e60 con 0x7fb2dc132f50 2023-07-02T09:01:33.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.572+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc06f490 con 0x7fb2dc132f50 2023-07-02T09:01:33.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.573+0000 7fb2d2ffd700 1 -- 172.21.15.103:0/1159625722 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb2cc0070b0 con 0x7fb2dc132f50 2023-07-02T09:01:33.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.573+0000 7fb2d2ffd700 1 --2- 172.21.15.103:0/1159625722 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb2c00377c0 0x7fb2c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:33.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.573+0000 7fb2d2ffd700 1 -- 172.21.15.103:0/1159625722 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb2cc0469f0 con 0x7fb2dc132f50 2023-07-02T09:01:33.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.573+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc06f490 con 0x7fb2dc132f50 2023-07-02T09:01:33.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.573+0000 7fb2e1831700 1 --2- 172.21.15.103:0/1159625722 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb2c00377c0 0x7fb2c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:33.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.574+0000 7fb2e1831700 1 --2- 172.21.15.103:0/1159625722 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb2c00377c0 0x7fb2c0039cd0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:33.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.578+0000 7fb2d2ffd700 1 -- 172.21.15.103:0/1159625722 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb2dc06f490 con 0x7fb2dc132f50 2023-07-02T09:01:33.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.770+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb2dc06f490 con 0x7fb2dc132f50 2023-07-02T09:01:33.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.770+0000 7fb2d2ffd700 1 -- 172.21.15.103:0/1159625722 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb2dc06f490 con 0x7fb2dc132f50 2023-07-02T09:01:33.773 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:33.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb2c00377c0 msgr2=0x7fb2c0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:33.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 --2- 172.21.15.103:0/1159625722 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb2c00377c0 0x7fb2c0039cd0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:33.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc132f50 msgr2=0x7fb2dc12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:33.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 --2- 172.21.15.103:0/1159625722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc132f50 0x7fb2dc12bea0 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:33.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 shutdown_connections 2023-07-02T09:01:33.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 --2- 172.21.15.103:0/1159625722 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb2c00377c0 0x7fb2c0039cd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:33.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 --2- 172.21.15.103:0/1159625722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2dc132f50 0x7fb2dc12bea0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:33.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 >> 172.21.15.103:0/1159625722 conn(0x7fb2dc0a2080 msgr2=0x7fb2dc1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:33.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 shutdown_connections 2023-07-02T09:01:33.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:33.773+0000 7fb2e4296700 1 -- 172.21.15.103:0/1159625722 wait complete. 2023-07-02T09:01:33.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:34.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:34.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.925+0000 7f00cce93700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:34.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.930+0000 7f00cce93700 1 Processor -- start 2023-07-02T09:01:34.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.930+0000 7f00cce93700 1 -- start start 2023-07-02T09:01:34.932 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00cce93700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00c812f250 0x7f00c812f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:34.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00cce93700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c806df60 con 0x7f00c8098a00 2023-07-02T09:01:34.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00cce93700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c809a710 con 0x7f00c812f250 2023-07-02T09:01:34.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00c812f250 0x7f00c812f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:34.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c5d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f00c8098a00 0x7f00c8098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37080/0 (socket says 172.21.15.103:37080) 2023-07-02T09:01:34.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00c812f250 0x7f00c812f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43886/0 (socket says 172.21.15.103:43886) 2023-07-02T09:01:34.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c5d9b700 1 -- 172.21.15.103:0/644457649 learned_addr learned my addr 172.21.15.103:0/644457649 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:34.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c659c700 1 -- 172.21.15.103:0/644457649 >> v1:172.21.15.103:6789/0 conn(0x7f00c8098a00 legacy=0x7f00c8098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:34.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c659c700 1 -- 172.21.15.103:0/644457649 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c812ed20 con 0x7f00c812f250 2023-07-02T09:01:34.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c659c700 1 --2- 172.21.15.103:0/644457649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c812f250 0x7f00c812f660 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53dc6900079dcb9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:34.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c559a700 1 -- 172.21.15.103:0/644457649 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f00bc009cd0 con 0x7f00c812f250 2023-07-02T09:01:34.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c559a700 1 -- 172.21.15.103:0/644457649 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f00bc009e30 con 0x7f00c812f250 2023-07-02T09:01:34.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.931+0000 7f00c559a700 1 -- 172.21.15.103:0/644457649 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f00bc00e360 con 0x7f00c812f250 2023-07-02T09:01:34.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.932+0000 7f00cce93700 1 -- 172.21.15.103:0/644457649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c812f250 msgr2=0x7f00c812f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:34.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.932+0000 7f00cce93700 1 --2- 172.21.15.103:0/644457649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c812f250 0x7f00c812f660 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:34.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.932+0000 7f00cce93700 1 -- 172.21.15.103:0/644457649 shutdown_connections 2023-07-02T09:01:34.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.932+0000 7f00cce93700 1 --2- 172.21.15.103:0/644457649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c812f250 0x7f00c812f660 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:34.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.932+0000 7f00cce93700 1 -- 172.21.15.103:0/644457649 >> 172.21.15.103:0/644457649 conn(0x7f00c809d810 msgr2=0x7f00c809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:34.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.932+0000 7f00cce93700 1 -- 172.21.15.103:0/644457649 shutdown_connections 2023-07-02T09:01:34.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.932+0000 7f00cce93700 1 -- 172.21.15.103:0/644457649 wait complete. 2023-07-02T09:01:34.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.932+0000 7f00cce93700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:34.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.933+0000 7f00cce93700 1 Processor -- start 2023-07-02T09:01:34.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.934+0000 7f00cce93700 1 -- start start 2023-07-02T09:01:34.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.934+0000 7f00cce93700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c8098a00 0x7f00c81c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:34.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.934+0000 7f00cce93700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c8117600 con 0x7f00c8098a00 2023-07-02T09:01:34.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.934+0000 7f00c659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c8098a00 0x7f00c81c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:34.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.934+0000 7f00c659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c8098a00 0x7f00c81c52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43896/0 (socket says 172.21.15.103:43896) 2023-07-02T09:01:34.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.934+0000 7f00c659c700 1 -- 172.21.15.103:0/5342921 learned_addr learned my addr 172.21.15.103:0/5342921 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:34.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.935+0000 7f00c659c700 1 -- 172.21.15.103:0/5342921 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c8118cb0 con 0x7f00c8098a00 2023-07-02T09:01:34.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.935+0000 7f00c659c700 1 --2- 172.21.15.103:0/5342921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c8098a00 0x7f00c81c52b0 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:34.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.935+0000 7f00b6ffd700 1 -- 172.21.15.103:0/5342921 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f00bc013530 con 0x7f00c8098a00 2023-07-02T09:01:34.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.935+0000 7f00b6ffd700 1 -- 172.21.15.103:0/5342921 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f00bc013690 con 0x7f00c8098a00 2023-07-02T09:01:34.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.935+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c8119af0 con 0x7f00c8098a00 2023-07-02T09:01:34.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.935+0000 7f00b6ffd700 1 -- 172.21.15.103:0/5342921 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f00bc013930 con 0x7f00c8098a00 2023-07-02T09:01:34.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.935+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c812c490 con 0x7f00c8098a00 2023-07-02T09:01:34.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.936+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c8117f20 con 0x7f00c8098a00 2023-07-02T09:01:34.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.936+0000 7f00b6ffd700 1 -- 172.21.15.103:0/5342921 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f00bc013ae0 con 0x7f00c8098a00 2023-07-02T09:01:34.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.936+0000 7f00b6ffd700 1 --2- 172.21.15.103:0/5342921 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f00a80377c0 0x7f00a8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:34.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.936+0000 7f00b6ffd700 1 -- 172.21.15.103:0/5342921 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f00bc018040 con 0x7f00c8098a00 2023-07-02T09:01:34.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.937+0000 7f00c5d9b700 1 --2- 172.21.15.103:0/5342921 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f00a80377c0 0x7f00a8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:34.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.937+0000 7f00c5d9b700 1 --2- 172.21.15.103:0/5342921 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f00a80377c0 0x7f00a8039cd0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:34.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:34.941+0000 7f00b6ffd700 1 -- 172.21.15.103:0/5342921 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f00bc018e00 con 0x7f00c8098a00 2023-07-02T09:01:35.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.132+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f00c8061ea0 con 0x7f00c8098a00 2023-07-02T09:01:35.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.132+0000 7f00b6ffd700 1 -- 172.21.15.103:0/5342921 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f00bc010e60 con 0x7f00c8098a00 2023-07-02T09:01:35.134 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:35.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f00a80377c0 msgr2=0x7f00a8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:35.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 --2- 172.21.15.103:0/5342921 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f00a80377c0 0x7f00a8039cd0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:35.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c8098a00 msgr2=0x7f00c81c52b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:35.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 --2- 172.21.15.103:0/5342921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c8098a00 0x7f00c81c52b0 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:35.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 shutdown_connections 2023-07-02T09:01:35.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 --2- 172.21.15.103:0/5342921 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f00a80377c0 0x7f00a8039cd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:35.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 --2- 172.21.15.103:0/5342921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c8098a00 0x7f00c81c52b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:35.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 >> 172.21.15.103:0/5342921 conn(0x7f00c809d810 msgr2=0x7f00c809f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:35.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 shutdown_connections 2023-07-02T09:01:35.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:35.134+0000 7f00cce93700 1 -- 172.21.15.103:0/5342921 wait complete. 2023-07-02T09:01:35.148 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:35.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~110s 2023-07-02T09:01:36.150 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:36.287 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.282+0000 7f5fbc848700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:36.288 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.286+0000 7f5fbc848700 1 Processor -- start 2023-07-02T09:01:36.288 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.286+0000 7f5fbc848700 1 -- start start 2023-07-02T09:01:36.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.287+0000 7f5fbc848700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5fb8132030 0x7f5fb8134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:36.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.287+0000 7f5fbc848700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb806df60 con 0x7f5fb812f250 2023-07-02T09:01:36.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.287+0000 7f5fbc848700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb809a710 con 0x7f5fb8132030 2023-07-02T09:01:36.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.287+0000 7f5fb659c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f5fb812f250 0x7f5fb812f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37086/0 (socket says 172.21.15.103:37086) 2023-07-02T09:01:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.287+0000 7f5fb5d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5fb8132030 0x7f5fb8134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.287+0000 7f5fb659c700 1 -- 172.21.15.103:0/4267755275 learned_addr learned my addr 172.21.15.103:0/4267755275 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.288+0000 7f5fb5d9b700 1 -- 172.21.15.103:0/4267755275 >> v1:172.21.15.103:6789/0 conn(0x7f5fb812f250 legacy=0x7f5fb812f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.288+0000 7f5fb5d9b700 1 -- 172.21.15.103:0/4267755275 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb812ed20 con 0x7f5fb8132030 2023-07-02T09:01:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.288+0000 7f5fb5d9b700 1 --2- 172.21.15.103:0/4267755275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb8132030 0x7f5fb8134450 crc :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da65d4003ceefa28 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.288+0000 7f5fb559a700 1 -- 172.21.15.103:0/4267755275 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5fa8009cd0 con 0x7f5fb8132030 2023-07-02T09:01:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.288+0000 7f5fb559a700 1 -- 172.21.15.103:0/4267755275 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5fa8009e30 con 0x7f5fb8132030 2023-07-02T09:01:36.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.288+0000 7f5fb559a700 1 -- 172.21.15.103:0/4267755275 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5fa800e360 con 0x7f5fb8132030 2023-07-02T09:01:36.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.289+0000 7f5fbc848700 1 -- 172.21.15.103:0/4267755275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb8132030 msgr2=0x7f5fb8134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:36.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.289+0000 7f5fbc848700 1 --2- 172.21.15.103:0/4267755275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb8132030 0x7f5fb8134450 crc :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:36.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.289+0000 7f5fbc848700 1 -- 172.21.15.103:0/4267755275 shutdown_connections 2023-07-02T09:01:36.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.289+0000 7f5fbc848700 1 --2- 172.21.15.103:0/4267755275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb8132030 0x7f5fb8134450 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:36.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.289+0000 7f5fbc848700 1 -- 172.21.15.103:0/4267755275 >> 172.21.15.103:0/4267755275 conn(0x7f5fb809d810 msgr2=0x7f5fb809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:36.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.289+0000 7f5fbc848700 1 -- 172.21.15.103:0/4267755275 shutdown_connections 2023-07-02T09:01:36.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.289+0000 7f5fbc848700 1 -- 172.21.15.103:0/4267755275 wait complete. 2023-07-02T09:01:36.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.289+0000 7f5fbc848700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:36.293 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.290+0000 7f5fbc848700 1 Processor -- start 2023-07-02T09:01:36.293 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.290+0000 7f5fbc848700 1 -- start start 2023-07-02T09:01:36.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fbc848700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb812f250 0x7f5fb81c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:36.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fbc848700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb811b9d0 con 0x7f5fb812f250 2023-07-02T09:01:36.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fb659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb812f250 0x7f5fb81c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:36.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fb659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb812f250 0x7f5fb81c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43920/0 (socket says 172.21.15.103:43920) 2023-07-02T09:01:36.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fb659c700 1 -- 172.21.15.103:0/2897195251 learned_addr learned my addr 172.21.15.103:0/2897195251 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:36.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fb659c700 1 -- 172.21.15.103:0/2897195251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb8071340 con 0x7f5fb812f250 2023-07-02T09:01:36.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fb659c700 1 --2- 172.21.15.103:0/2897195251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb812f250 0x7f5fb81c7250 crc :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:36.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fa6ffd700 1 -- 172.21.15.103:0/2897195251 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5fac0061e0 con 0x7f5fb812f250 2023-07-02T09:01:36.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fa6ffd700 1 -- 172.21.15.103:0/2897195251 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5fac006340 con 0x7f5fb812f250 2023-07-02T09:01:36.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb8071d60 con 0x7f5fb812f250 2023-07-02T09:01:36.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fa6ffd700 1 -- 172.21.15.103:0/2897195251 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5fac0065e0 con 0x7f5fb812f250 2023-07-02T09:01:36.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.291+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb8129060 con 0x7f5fb812f250 2023-07-02T09:01:36.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.292+0000 7f5fa6ffd700 1 -- 172.21.15.103:0/2897195251 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f5fac004840 con 0x7f5fb812f250 2023-07-02T09:01:36.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.292+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb811bb10 con 0x7f5fb812f250 2023-07-02T09:01:36.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.292+0000 7f5fa6ffd700 1 --2- 172.21.15.103:0/2897195251 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5f980377c0 0x7f5f98039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:36.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.292+0000 7f5fa6ffd700 1 -- 172.21.15.103:0/2897195251 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f5fac046b40 con 0x7f5fb812f250 2023-07-02T09:01:36.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.293+0000 7f5fb5d9b700 1 --2- 172.21.15.103:0/2897195251 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5f980377c0 0x7f5f98039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:36.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.293+0000 7f5fb5d9b700 1 --2- 172.21.15.103:0/2897195251 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5f980377c0 0x7f5f98039cd0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:36.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.297+0000 7f5fa6ffd700 1 -- 172.21.15.103:0/2897195251 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f5fac00a6e0 con 0x7f5fb812f250 2023-07-02T09:01:36.490 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.488+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5fb8061ea0 con 0x7f5fb812f250 2023-07-02T09:01:36.490 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.488+0000 7f5fa6ffd700 1 -- 172.21.15.103:0/2897195251 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f5fac00a120 con 0x7f5fb812f250 2023-07-02T09:01:36.490 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:36.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.490+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5f980377c0 msgr2=0x7f5f98039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:36.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.490+0000 7f5fbc848700 1 --2- 172.21.15.103:0/2897195251 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5f980377c0 0x7f5f98039cd0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:36.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.490+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb812f250 msgr2=0x7f5fb81c7250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:36.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.490+0000 7f5fbc848700 1 --2- 172.21.15.103:0/2897195251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb812f250 0x7f5fb81c7250 crc :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:36.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.491+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 shutdown_connections 2023-07-02T09:01:36.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.491+0000 7f5fbc848700 1 --2- 172.21.15.103:0/2897195251 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f5f980377c0 0x7f5f98039cd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:36.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.491+0000 7f5fbc848700 1 --2- 172.21.15.103:0/2897195251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fb812f250 0x7f5fb81c7250 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:36.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.491+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 >> 172.21.15.103:0/2897195251 conn(0x7f5fb809d810 msgr2=0x7f5fb812e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:36.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.491+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 shutdown_connections 2023-07-02T09:01:36.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:36.491+0000 7f5fbc848700 1 -- 172.21.15.103:0/2897195251 wait complete. 2023-07-02T09:01:36.505 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:37.505 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:37.649 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.644+0000 7f7fbacb8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:37.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.649+0000 7f7fbacb8700 1 Processor -- start 2023-07-02T09:01:37.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.649+0000 7f7fbacb8700 1 -- start start 2023-07-02T09:01:37.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.649+0000 7f7fbacb8700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7fb4132030 0x7f7fb4134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:37.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.649+0000 7f7fbacb8700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb406df60 con 0x7f7fb412f250 2023-07-02T09:01:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.649+0000 7f7fbacb8700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb409a710 con 0x7f7fb4132030 2023-07-02T09:01:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb3fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7fb4132030 0x7f7fb4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb8a54700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7fb412f250 0x7f7fb412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37090/0 (socket says 172.21.15.103:37090) 2023-07-02T09:01:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb3fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7fb4132030 0x7f7fb4134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43932/0 (socket says 172.21.15.103:43932) 2023-07-02T09:01:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb8a54700 1 -- 172.21.15.103:0/3754647228 learned_addr learned my addr 172.21.15.103:0/3754647228 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb3fff700 1 -- 172.21.15.103:0/3754647228 >> v1:172.21.15.103:6789/0 conn(0x7f7fb412f250 legacy=0x7f7fb412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb3fff700 1 -- 172.21.15.103:0/3754647228 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb412ed20 con 0x7f7fb4132030 2023-07-02T09:01:37.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb3fff700 1 --2- 172.21.15.103:0/3754647228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb4132030 0x7f7fb4134450 crc :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60aa2a0384b19ae9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:37.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb37fe700 1 -- 172.21.15.103:0/3754647228 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7fa000d450 con 0x7f7fb4132030 2023-07-02T09:01:37.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb37fe700 1 -- 172.21.15.103:0/3754647228 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7fa000d5b0 con 0x7f7fb4132030 2023-07-02T09:01:37.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.650+0000 7f7fb37fe700 1 -- 172.21.15.103:0/3754647228 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7fa000d850 con 0x7f7fb4132030 2023-07-02T09:01:37.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.651+0000 7f7fbacb8700 1 -- 172.21.15.103:0/3754647228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb4132030 msgr2=0x7f7fb4134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:37.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.651+0000 7f7fbacb8700 1 --2- 172.21.15.103:0/3754647228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb4132030 0x7f7fb4134450 crc :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:37.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.651+0000 7f7fbacb8700 1 -- 172.21.15.103:0/3754647228 shutdown_connections 2023-07-02T09:01:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.651+0000 7f7fbacb8700 1 --2- 172.21.15.103:0/3754647228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb4132030 0x7f7fb4134450 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.651+0000 7f7fbacb8700 1 -- 172.21.15.103:0/3754647228 >> 172.21.15.103:0/3754647228 conn(0x7f7fb409d810 msgr2=0x7f7fb409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.651+0000 7f7fbacb8700 1 -- 172.21.15.103:0/3754647228 shutdown_connections 2023-07-02T09:01:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.651+0000 7f7fbacb8700 1 -- 172.21.15.103:0/3754647228 wait complete. 2023-07-02T09:01:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.651+0000 7f7fbacb8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.652+0000 7f7fbacb8700 1 Processor -- start 2023-07-02T09:01:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.652+0000 7f7fbacb8700 1 -- start start 2023-07-02T09:01:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.653+0000 7f7fbacb8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb412f250 0x7f7fb4070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:37.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.653+0000 7f7fbacb8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb4071240 con 0x7f7fb412f250 2023-07-02T09:01:37.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.653+0000 7f7fb8a54700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb412f250 0x7f7fb4070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:37.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.653+0000 7f7fb8a54700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb412f250 0x7f7fb4070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43940/0 (socket says 172.21.15.103:43940) 2023-07-02T09:01:37.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.653+0000 7f7fb8a54700 1 -- 172.21.15.103:0/327257466 learned_addr learned my addr 172.21.15.103:0/327257466 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:37.657 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.653+0000 7f7fb8a54700 1 -- 172.21.15.103:0/327257466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb4119220 con 0x7f7fb412f250 2023-07-02T09:01:37.657 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.653+0000 7f7fb8a54700 1 --2- 172.21.15.103:0/327257466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb412f250 0x7f7fb4070100 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.654+0000 7f7fb17fa700 1 -- 172.21.15.103:0/327257466 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7fa8005a60 con 0x7f7fb412f250 2023-07-02T09:01:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.654+0000 7f7fb17fa700 1 -- 172.21.15.103:0/327257466 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7fa8005bc0 con 0x7f7fb412f250 2023-07-02T09:01:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.654+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb4071d60 con 0x7f7fb412f250 2023-07-02T09:01:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.654+0000 7f7fb17fa700 1 -- 172.21.15.103:0/327257466 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7fa8005e60 con 0x7f7fb412f250 2023-07-02T09:01:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.654+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb4118d50 con 0x7f7fb412f250 2023-07-02T09:01:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.654+0000 7f7fb17fa700 1 -- 172.21.15.103:0/327257466 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7fa80070b0 con 0x7f7fb412f250 2023-07-02T09:01:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.655+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb4061ea0 con 0x7f7fb412f250 2023-07-02T09:01:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.655+0000 7f7fb17fa700 1 --2- 172.21.15.103:0/327257466 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7f980377c0 0x7f7f98039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:37.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.655+0000 7f7fb17fa700 1 -- 172.21.15.103:0/327257466 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7fa8046b30 con 0x7f7fb412f250 2023-07-02T09:01:37.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.655+0000 7f7fb3fff700 1 --2- 172.21.15.103:0/327257466 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7f980377c0 0x7f7f98039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:37.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.655+0000 7f7fb3fff700 1 --2- 172.21.15.103:0/327257466 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7f980377c0 0x7f7f98039cd0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:37.660 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.659+0000 7f7fb17fa700 1 -- 172.21.15.103:0/327257466 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7fa800fdf0 con 0x7f7fb412f250 2023-07-02T09:01:37.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.853+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7fb412c6c0 con 0x7f7fb412f250 2023-07-02T09:01:37.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.853+0000 7f7fb17fa700 1 -- 172.21.15.103:0/327257466 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7fa800ddd0 con 0x7f7fb412f250 2023-07-02T09:01:37.855 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:37.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7f980377c0 msgr2=0x7f7f98039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 --2- 172.21.15.103:0/327257466 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7f980377c0 0x7f7f98039cd0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb412f250 msgr2=0x7f7fb4070100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 --2- 172.21.15.103:0/327257466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb412f250 0x7f7fb4070100 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 shutdown_connections 2023-07-02T09:01:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 --2- 172.21.15.103:0/327257466 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7f980377c0 0x7f7f98039cd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 --2- 172.21.15.103:0/327257466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb412f250 0x7f7fb4070100 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:37.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 >> 172.21.15.103:0/327257466 conn(0x7f7fb409d810 msgr2=0x7f7fb412e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:37.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 shutdown_connections 2023-07-02T09:01:37.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:37.855+0000 7f7fbacb8700 1 -- 172.21.15.103:0/327257466 wait complete. 2023-07-02T09:01:37.870 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:38.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:39.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.008+0000 7f39b7499700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:39.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.013+0000 7f39b7499700 1 Processor -- start 2023-07-02T09:01:39.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.013+0000 7f39b7499700 1 -- start start 2023-07-02T09:01:39.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.014+0000 7f39b7499700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f39b0070240 0x7f39b0135c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:39.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.014+0000 7f39b7499700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0067780 con 0x7f39b006f720 2023-07-02T09:01:39.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.014+0000 7f39b7499700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b0067f80 con 0x7f39b0070240 2023-07-02T09:01:39.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.014+0000 7f39b4a34700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f39b0070240 0x7f39b0135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.014+0000 7f39b5235700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f39b006f720 0x7f39b006fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37092/0 (socket says 172.21.15.103:37092) 2023-07-02T09:01:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.014+0000 7f39b5235700 1 -- 172.21.15.103:0/2668152104 learned_addr learned my addr 172.21.15.103:0/2668152104 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39b4a34700 1 -- 172.21.15.103:0/2668152104 >> v1:172.21.15.103:6789/0 conn(0x7f39b006f720 legacy=0x7f39b006fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39a7fff700 1 -- 172.21.15.103:0/2668152104 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760832408 0 0) 0x7f39b0067780 con 0x7f39b006f720 2023-07-02T09:01:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39b4a34700 1 -- 172.21.15.103:0/2668152104 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b0093f30 con 0x7f39b0070240 2023-07-02T09:01:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39b4a34700 1 --2- 172.21.15.103:0/2668152104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b0070240 0x7f39b0135c40 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1cc86cf0e410bf server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39a7fff700 1 -- 172.21.15.103:0/2668152104 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f39ac009cd0 con 0x7f39b0070240 2023-07-02T09:01:39.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39a7fff700 1 -- 172.21.15.103:0/2668152104 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f39ac009e30 con 0x7f39b0070240 2023-07-02T09:01:39.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39a7fff700 1 -- 172.21.15.103:0/2668152104 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f39ac00e360 con 0x7f39b0070240 2023-07-02T09:01:39.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39b7499700 1 -- 172.21.15.103:0/2668152104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b0070240 msgr2=0x7f39b0135c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39b7499700 1 --2- 172.21.15.103:0/2668152104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b0070240 0x7f39b0135c40 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39b7499700 1 -- 172.21.15.103:0/2668152104 shutdown_connections 2023-07-02T09:01:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.015+0000 7f39b7499700 1 --2- 172.21.15.103:0/2668152104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b0070240 0x7f39b0135c40 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.016+0000 7f39b7499700 1 -- 172.21.15.103:0/2668152104 >> 172.21.15.103:0/2668152104 conn(0x7f39b00a22a0 msgr2=0x7f39b006d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.016+0000 7f39b7499700 1 -- 172.21.15.103:0/2668152104 shutdown_connections 2023-07-02T09:01:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.016+0000 7f39b7499700 1 -- 172.21.15.103:0/2668152104 wait complete. 2023-07-02T09:01:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.016+0000 7f39b7499700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.017+0000 7f39b7499700 1 Processor -- start 2023-07-02T09:01:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.017+0000 7f39b7499700 1 -- start start 2023-07-02T09:01:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.017+0000 7f39b7499700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b006f720 0x7f39b0098fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.017+0000 7f39b7499700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b009b2e0 con 0x7f39b006f720 2023-07-02T09:01:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39b5235700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b006f720 0x7f39b0098fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39b5235700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b006f720 0x7f39b0098fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43968/0 (socket says 172.21.15.103:43968) 2023-07-02T09:01:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39b5235700 1 -- 172.21.15.103:0/302820834 learned_addr learned my addr 172.21.15.103:0/302820834 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39b5235700 1 -- 172.21.15.103:0/302820834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b009c960 con 0x7f39b006f720 2023-07-02T09:01:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39b5235700 1 --2- 172.21.15.103:0/302820834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b006f720 0x7f39b0098fb0 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39a5ffb700 1 -- 172.21.15.103:0/302820834 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f39a0004b80 con 0x7f39b006f720 2023-07-02T09:01:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39a5ffb700 1 -- 172.21.15.103:0/302820834 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f39a0004ce0 con 0x7f39b006f720 2023-07-02T09:01:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b0098c10 con 0x7f39b006f720 2023-07-02T09:01:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39a5ffb700 1 -- 172.21.15.103:0/302820834 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f39a000c550 con 0x7f39b006f720 2023-07-02T09:01:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.018+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b0098820 con 0x7f39b006f720 2023-07-02T09:01:39.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.019+0000 7f39a5ffb700 1 -- 172.21.15.103:0/302820834 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f39a000c6b0 con 0x7f39b006f720 2023-07-02T09:01:39.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.019+0000 7f39a5ffb700 1 --2- 172.21.15.103:0/302820834 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f39940377c0 0x7f3994039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:39.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.019+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b0098e00 con 0x7f39b006f720 2023-07-02T09:01:39.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.019+0000 7f39a5ffb700 1 -- 172.21.15.103:0/302820834 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f39a0046a60 con 0x7f39b006f720 2023-07-02T09:01:39.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.020+0000 7f39b4a34700 1 --2- 172.21.15.103:0/302820834 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f39940377c0 0x7f3994039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:39.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.020+0000 7f39b4a34700 1 --2- 172.21.15.103:0/302820834 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f39940377c0 0x7f3994039cd0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:39.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.024+0000 7f39a5ffb700 1 -- 172.21.15.103:0/302820834 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f39a0009c80 con 0x7f39b006f720 2023-07-02T09:01:39.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.215+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f39b0061ea0 con 0x7f39b006f720 2023-07-02T09:01:39.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.216+0000 7f39a5ffb700 1 -- 172.21.15.103:0/302820834 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f39a00098a0 con 0x7f39b006f720 2023-07-02T09:01:39.217 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:39.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f39940377c0 msgr2=0x7f3994039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:39.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 --2- 172.21.15.103:0/302820834 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f39940377c0 0x7f3994039cd0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:39.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b006f720 msgr2=0x7f39b0098fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 --2- 172.21.15.103:0/302820834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b006f720 0x7f39b0098fb0 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 shutdown_connections 2023-07-02T09:01:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 --2- 172.21.15.103:0/302820834 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f39940377c0 0x7f3994039cd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 --2- 172.21.15.103:0/302820834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b006f720 0x7f39b0098fb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 >> 172.21.15.103:0/302820834 conn(0x7f39b00a22a0 msgr2=0x7f39b008f670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 shutdown_connections 2023-07-02T09:01:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:39.218+0000 7f39b7499700 1 -- 172.21.15.103:0/302820834 wait complete. 2023-07-02T09:01:39.232 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:40.233 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.364+0000 7f9e86588700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.369+0000 7f9e86588700 1 Processor -- start 2023-07-02T09:01:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.369+0000 7f9e86588700 1 -- start start 2023-07-02T09:01:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e86588700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9e80133d60 0x7f9e80136180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e86588700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e80130a50 con 0x7f9e80130f30 2023-07-02T09:01:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e86588700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8009b650 con 0x7f9e80133d60 2023-07-02T09:01:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e7f7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9e80133d60 0x7f9e80136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e7f7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9e80133d60 0x7f9e80136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43984/0 (socket says 172.21.15.103:43984) 2023-07-02T09:01:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e7ffff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9e80130f30 0x7f9e80131340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37098/0 (socket says 172.21.15.103:37098) 2023-07-02T09:01:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e7f7fe700 1 -- 172.21.15.103:0/432697336 learned_addr learned my addr 172.21.15.103:0/432697336 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e7f7fe700 1 -- 172.21.15.103:0/432697336 >> v1:172.21.15.103:6789/0 conn(0x7f9e80130f30 legacy=0x7f9e80131340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e7f7fe700 1 -- 172.21.15.103:0/432697336 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8009b8f0 con 0x7f9e80133d60 2023-07-02T09:01:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e7f7fe700 1 --2- 172.21.15.103:0/432697336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80133d60 0x7f9e80136180 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86b114618cc9a97 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.370+0000 7f9e7effd700 1 -- 172.21.15.103:0/432697336 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9e70009d90 con 0x7f9e80133d60 2023-07-02T09:01:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e7effd700 1 -- 172.21.15.103:0/432697336 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9e7000e230 con 0x7f9e80133d60 2023-07-02T09:01:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e7effd700 1 -- 172.21.15.103:0/432697336 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9e7000e4d0 con 0x7f9e80133d60 2023-07-02T09:01:40.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e86588700 1 -- 172.21.15.103:0/432697336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80133d60 msgr2=0x7f9e80136180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:40.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e86588700 1 --2- 172.21.15.103:0/432697336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80133d60 0x7f9e80136180 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:40.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e86588700 1 -- 172.21.15.103:0/432697336 shutdown_connections 2023-07-02T09:01:40.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e86588700 1 --2- 172.21.15.103:0/432697336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80133d60 0x7f9e80136180 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e86588700 1 -- 172.21.15.103:0/432697336 >> 172.21.15.103:0/432697336 conn(0x7f9e800a0100 msgr2=0x7f9e800a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e86588700 1 -- 172.21.15.103:0/432697336 shutdown_connections 2023-07-02T09:01:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e86588700 1 -- 172.21.15.103:0/432697336 wait complete. 2023-07-02T09:01:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.371+0000 7f9e86588700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.372+0000 7f9e86588700 1 Processor -- start 2023-07-02T09:01:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.373+0000 7f9e86588700 1 -- start start 2023-07-02T09:01:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.373+0000 7f9e86588700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80130f30 0x7f9e8011a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.373+0000 7f9e86588700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8012de40 con 0x7f9e80130f30 2023-07-02T09:01:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.373+0000 7f9e7ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80130f30 0x7f9e8011a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.373+0000 7f9e7ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80130f30 0x7f9e8011a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43992/0 (socket says 172.21.15.103:43992) 2023-07-02T09:01:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.373+0000 7f9e7ffff700 1 -- 172.21.15.103:0/2631971482 learned_addr learned my addr 172.21.15.103:0/2631971482 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.374+0000 7f9e7ffff700 1 -- 172.21.15.103:0/2631971482 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8011da80 con 0x7f9e80130f30 2023-07-02T09:01:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.374+0000 7f9e7ffff700 1 --2- 172.21.15.103:0/2631971482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80130f30 0x7f9e8011a4a0 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.374+0000 7f9e7cff9700 1 -- 172.21.15.103:0/2631971482 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9e740071e0 con 0x7f9e80130f30 2023-07-02T09:01:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.374+0000 7f9e7cff9700 1 -- 172.21.15.103:0/2631971482 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9e74007340 con 0x7f9e80130f30 2023-07-02T09:01:40.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.374+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8011c100 con 0x7f9e80130f30 2023-07-02T09:01:40.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.374+0000 7f9e7cff9700 1 -- 172.21.15.103:0/2631971482 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9e740075e0 con 0x7f9e80130f30 2023-07-02T09:01:40.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.374+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8011e5f0 con 0x7f9e80130f30 2023-07-02T09:01:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.375+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8004e800 con 0x7f9e80130f30 2023-07-02T09:01:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.375+0000 7f9e7cff9700 1 -- 172.21.15.103:0/2631971482 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f9e74005800 con 0x7f9e80130f30 2023-07-02T09:01:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.375+0000 7f9e7cff9700 1 --2- 172.21.15.103:0/2631971482 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9e6c0377b0 0x7f9e6c039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.375+0000 7f9e7cff9700 1 -- 172.21.15.103:0/2631971482 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9e74046be0 con 0x7f9e80130f30 2023-07-02T09:01:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.375+0000 7f9e7f7fe700 1 --2- 172.21.15.103:0/2631971482 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9e6c0377b0 0x7f9e6c039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.376+0000 7f9e7f7fe700 1 --2- 172.21.15.103:0/2631971482 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9e6c0377b0 0x7f9e6c039cc0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:40.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.380+0000 7f9e7cff9700 1 -- 172.21.15.103:0/2631971482 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f9e7400ee40 con 0x7f9e80130f30 2023-07-02T09:01:40.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.569+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9e8011d8a0 con 0x7f9e80130f30 2023-07-02T09:01:40.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.570+0000 7f9e7cff9700 1 -- 172.21.15.103:0/2631971482 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f9e7400ee40 con 0x7f9e80130f30 2023-07-02T09:01:40.571 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9e6c0377b0 msgr2=0x7f9e6c039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 --2- 172.21.15.103:0/2631971482 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9e6c0377b0 0x7f9e6c039cc0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80130f30 msgr2=0x7f9e8011a4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 --2- 172.21.15.103:0/2631971482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80130f30 0x7f9e8011a4a0 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 shutdown_connections 2023-07-02T09:01:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 --2- 172.21.15.103:0/2631971482 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9e6c0377b0 0x7f9e6c039cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 --2- 172.21.15.103:0/2631971482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e80130f30 0x7f9e8011a4a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 >> 172.21.15.103:0/2631971482 conn(0x7f9e800a0100 msgr2=0x7f9e80131340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 shutdown_connections 2023-07-02T09:01:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:40.572+0000 7f9e86588700 1 -- 172.21.15.103:0/2631971482 wait complete. 2023-07-02T09:01:40.586 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:41.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~116s 2023-07-02T09:01:41.587 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.725+0000 7f331855c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.730+0000 7f331855c700 1 Processor -- start 2023-07-02T09:01:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.730+0000 7f331855c700 1 -- start start 2023-07-02T09:01:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f331855c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f331006d6e0 0x7f3310099b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f331855c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331009cea0 con 0x7f331009a050 2023-07-02T09:01:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f331855c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331006d490 con 0x7f331006d6e0 2023-07-02T09:01:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f3315af7700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f331009a050 0x7f331012f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37114/0 (socket says 172.21.15.103:37114) 2023-07-02T09:01:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f33162f8700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f331006d6e0 0x7f3310099b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f3315af7700 1 -- 172.21.15.103:0/3873246101 learned_addr learned my addr 172.21.15.103:0/3873246101 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f33162f8700 1 -- 172.21.15.103:0/3873246101 >> v1:172.21.15.103:6789/0 conn(0x7f331009a050 legacy=0x7f331012f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f33152f6700 1 -- 172.21.15.103:0/3873246101 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607925278 0 0) 0x7f331009cea0 con 0x7f331009a050 2023-07-02T09:01:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.731+0000 7f33162f8700 1 -- 172.21.15.103:0/3873246101 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331006dc80 con 0x7f331006d6e0 2023-07-02T09:01:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f33162f8700 1 --2- 172.21.15.103:0/3873246101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f3310099b10 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5caf41f67d32a3 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f33152f6700 1 -- 172.21.15.103:0/3873246101 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3300009cd0 con 0x7f331006d6e0 2023-07-02T09:01:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f33152f6700 1 -- 172.21.15.103:0/3873246101 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3300009e30 con 0x7f331006d6e0 2023-07-02T09:01:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f33152f6700 1 -- 172.21.15.103:0/3873246101 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f330000e360 con 0x7f331006d6e0 2023-07-02T09:01:41.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f331855c700 1 -- 172.21.15.103:0/3873246101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 msgr2=0x7f3310099b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:41.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f331855c700 1 --2- 172.21.15.103:0/3873246101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f3310099b10 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:41.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f331855c700 1 -- 172.21.15.103:0/3873246101 shutdown_connections 2023-07-02T09:01:41.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f331855c700 1 --2- 172.21.15.103:0/3873246101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f3310099b10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f331855c700 1 -- 172.21.15.103:0/3873246101 >> 172.21.15.103:0/3873246101 conn(0x7f331009d810 msgr2=0x7f331009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f331855c700 1 -- 172.21.15.103:0/3873246101 shutdown_connections 2023-07-02T09:01:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.732+0000 7f331855c700 1 -- 172.21.15.103:0/3873246101 wait complete. 2023-07-02T09:01:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.733+0000 7f331855c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.733+0000 7f331855c700 1 Processor -- start 2023-07-02T09:01:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.734+0000 7f331855c700 1 -- start start 2023-07-02T09:01:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.734+0000 7f331855c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f33101c2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.734+0000 7f331855c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331012efc0 con 0x7f331006d6e0 2023-07-02T09:01:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.734+0000 7f33162f8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f33101c2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.734+0000 7f33162f8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f33101c2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44000/0 (socket says 172.21.15.103:44000) 2023-07-02T09:01:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.734+0000 7f33162f8700 1 -- 172.21.15.103:0/1712669650 learned_addr learned my addr 172.21.15.103:0/1712669650 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.735+0000 7f33162f8700 1 -- 172.21.15.103:0/1712669650 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331011fc40 con 0x7f331006d6e0 2023-07-02T09:01:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.735+0000 7f33162f8700 1 --2- 172.21.15.103:0/1712669650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f33101c2c50 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.735+0000 7f3306ffd700 1 -- 172.21.15.103:0/1712669650 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3300013530 con 0x7f331006d6e0 2023-07-02T09:01:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.735+0000 7f3306ffd700 1 -- 172.21.15.103:0/1712669650 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3300013690 con 0x7f331006d6e0 2023-07-02T09:01:41.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.735+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331011ec70 con 0x7f331006d6e0 2023-07-02T09:01:41.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.735+0000 7f3306ffd700 1 -- 172.21.15.103:0/1712669650 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3300013930 con 0x7f331006d6e0 2023-07-02T09:01:41.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.735+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33101b5750 con 0x7f331006d6e0 2023-07-02T09:01:41.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.736+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3310061ea0 con 0x7f331006d6e0 2023-07-02T09:01:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.736+0000 7f3306ffd700 1 -- 172.21.15.103:0/1712669650 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3300013ae0 con 0x7f331006d6e0 2023-07-02T09:01:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.736+0000 7f3306ffd700 1 --2- 172.21.15.103:0/1712669650 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32f40377b0 0x7f32f4039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.736+0000 7f3306ffd700 1 -- 172.21.15.103:0/1712669650 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3300018040 con 0x7f331006d6e0 2023-07-02T09:01:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.737+0000 7f3315af7700 1 --2- 172.21.15.103:0/1712669650 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32f40377b0 0x7f32f4039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.737+0000 7f3315af7700 1 --2- 172.21.15.103:0/1712669650 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32f40377b0 0x7f32f4039cc0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:41.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.741+0000 7f3306ffd700 1 -- 172.21.15.103:0/1712669650 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3300018a80 con 0x7f331006d6e0 2023-07-02T09:01:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.932+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f331011fc40 con 0x7f331006d6e0 2023-07-02T09:01:41.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.932+0000 7f3306ffd700 1 -- 172.21.15.103:0/1712669650 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f331011fc40 con 0x7f331006d6e0 2023-07-02T09:01:41.934 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32f40377b0 msgr2=0x7f32f4039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 --2- 172.21.15.103:0/1712669650 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32f40377b0 0x7f32f4039cc0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 msgr2=0x7f33101c2c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 --2- 172.21.15.103:0/1712669650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f33101c2c50 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 shutdown_connections 2023-07-02T09:01:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 --2- 172.21.15.103:0/1712669650 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f32f40377b0 0x7f32f4039cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 --2- 172.21.15.103:0/1712669650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331006d6e0 0x7f33101c2c50 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 >> 172.21.15.103:0/1712669650 conn(0x7f331009d810 msgr2=0x7f3310070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.934+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 shutdown_connections 2023-07-02T09:01:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:41.935+0000 7f331855c700 1 -- 172.21.15.103:0/1712669650 wait complete. 2023-07-02T09:01:41.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:42.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.088+0000 7f52a3046700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:43.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.093+0000 7f52a3046700 1 Processor -- start 2023-07-02T09:01:43.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.093+0000 7f52a3046700 1 -- start start 2023-07-02T09:01:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f52a3046700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f529c133360 0x7f529c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f52a3046700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529c09e260 con 0x7f529c132f50 2023-07-02T09:01:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f52a3046700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529c0a1640 con 0x7f529c133360 2023-07-02T09:01:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f529bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f529c133360 0x7f529c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f529bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f529c133360 0x7f529c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44008/0 (socket says 172.21.15.103:44008) 2023-07-02T09:01:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f52a0de2700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f529c132f50 0x7f529c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37116/0 (socket says 172.21.15.103:37116) 2023-07-02T09:01:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f529bfff700 1 -- 172.21.15.103:0/2199792401 learned_addr learned my addr 172.21.15.103:0/2199792401 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:43.097 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f529bfff700 1 -- 172.21.15.103:0/2199792401 >> v1:172.21.15.103:6789/0 conn(0x7f529c132f50 legacy=0x7f529c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:43.097 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f529bfff700 1 -- 172.21.15.103:0/2199792401 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c132a20 con 0x7f529c133360 2023-07-02T09:01:43.097 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f529bfff700 1 --2- 172.21.15.103:0/2199792401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c133360 0x7f529c09cc90 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb71ea2a59c5336e server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:43.097 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.094+0000 7f529b7fe700 1 -- 172.21.15.103:0/2199792401 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5288009d90 con 0x7f529c133360 2023-07-02T09:01:43.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f529b7fe700 1 -- 172.21.15.103:0/2199792401 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f528800e230 con 0x7f529c133360 2023-07-02T09:01:43.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f529b7fe700 1 -- 172.21.15.103:0/2199792401 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f528800e4d0 con 0x7f529c133360 2023-07-02T09:01:43.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f52a3046700 1 -- 172.21.15.103:0/2199792401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c133360 msgr2=0x7f529c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:43.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f52a3046700 1 --2- 172.21.15.103:0/2199792401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c133360 0x7f529c09cc90 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:43.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f52a3046700 1 -- 172.21.15.103:0/2199792401 shutdown_connections 2023-07-02T09:01:43.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f52a3046700 1 --2- 172.21.15.103:0/2199792401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c133360 0x7f529c09cc90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:43.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f52a3046700 1 -- 172.21.15.103:0/2199792401 >> 172.21.15.103:0/2199792401 conn(0x7f529c0a2080 msgr2=0x7f529c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:43.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f52a3046700 1 -- 172.21.15.103:0/2199792401 shutdown_connections 2023-07-02T09:01:43.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.095+0000 7f52a3046700 1 -- 172.21.15.103:0/2199792401 wait complete. 2023-07-02T09:01:43.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.096+0000 7f52a3046700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.097+0000 7f52a3046700 1 Processor -- start 2023-07-02T09:01:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.097+0000 7f52a3046700 1 -- start start 2023-07-02T09:01:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.097+0000 7f52a3046700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c132f50 0x7f529c12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.097+0000 7f52a3046700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c06d260 con 0x7f529c132f50 2023-07-02T09:01:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.097+0000 7f52a0de2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c132f50 0x7f529c12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.097+0000 7f52a0de2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c132f50 0x7f529c12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44016/0 (socket says 172.21.15.103:44016) 2023-07-02T09:01:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.097+0000 7f52a0de2700 1 -- 172.21.15.103:0/1356526974 learned_addr learned my addr 172.21.15.103:0/1356526974 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.098+0000 7f52a0de2700 1 -- 172.21.15.103:0/1356526974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c12bb90 con 0x7f529c132f50 2023-07-02T09:01:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.098+0000 7f52a0de2700 1 --2- 172.21.15.103:0/1356526974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c132f50 0x7f529c12bea0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:43.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.098+0000 7f52997fa700 1 -- 172.21.15.103:0/1356526974 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f52900059a0 con 0x7f529c132f50 2023-07-02T09:01:43.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.098+0000 7f52997fa700 1 -- 172.21.15.103:0/1356526974 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f5290005b00 con 0x7f529c132f50 2023-07-02T09:01:43.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.098+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529c06f730 con 0x7f529c132f50 2023-07-02T09:01:43.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.098+0000 7f52997fa700 1 -- 172.21.15.103:0/1356526974 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f5290005da0 con 0x7f529c132f50 2023-07-02T09:01:43.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.098+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529c06f490 con 0x7f529c132f50 2023-07-02T09:01:43.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.099+0000 7f52997fa700 1 -- 172.21.15.103:0/1356526974 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f52900070b0 con 0x7f529c132f50 2023-07-02T09:01:43.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.099+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529c061ea0 con 0x7f529c132f50 2023-07-02T09:01:43.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.099+0000 7f52997fa700 1 --2- 172.21.15.103:0/1356526974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52800377c0 0x7f5280039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:43.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.099+0000 7f52997fa700 1 -- 172.21.15.103:0/1356526974 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f5290046a70 con 0x7f529c132f50 2023-07-02T09:01:43.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.099+0000 7f529bfff700 1 --2- 172.21.15.103:0/1356526974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52800377c0 0x7f5280039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:43.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.100+0000 7f529bfff700 1 --2- 172.21.15.103:0/1356526974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52800377c0 0x7f5280039cd0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:43.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.104+0000 7f52997fa700 1 -- 172.21.15.103:0/1356526974 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f529000a6b0 con 0x7f529c132f50 2023-07-02T09:01:43.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.293+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f529c06f490 con 0x7f529c132f50 2023-07-02T09:01:43.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.294+0000 7f52997fa700 1 -- 172.21.15.103:0/1356526974 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f529c06f490 con 0x7f529c132f50 2023-07-02T09:01:43.295 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:43.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52800377c0 msgr2=0x7f5280039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:43.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 --2- 172.21.15.103:0/1356526974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52800377c0 0x7f5280039cd0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:43.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c132f50 msgr2=0x7f529c12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:43.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 --2- 172.21.15.103:0/1356526974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c132f50 0x7f529c12bea0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:43.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 shutdown_connections 2023-07-02T09:01:43.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 --2- 172.21.15.103:0/1356526974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52800377c0 0x7f5280039cd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:43.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 --2- 172.21.15.103:0/1356526974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c132f50 0x7f529c12bea0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:43.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 >> 172.21.15.103:0/1356526974 conn(0x7f529c0a2080 msgr2=0x7f529c1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:43.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 shutdown_connections 2023-07-02T09:01:43.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:43.296+0000 7f52a3046700 1 -- 172.21.15.103:0/1356526974 wait complete. 2023-07-02T09:01:43.310 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:44.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:44.455 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.449+0000 7fdcad3fd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:44.456 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.455+0000 7fdcad3fd700 1 Processor -- start 2023-07-02T09:01:44.456 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.455+0000 7fdcad3fd700 1 -- start start 2023-07-02T09:01:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.455+0000 7fdcad3fd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdca8133880 0x7fdca8135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.455+0000 7fdcad3fd700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca8071cc0 con 0x7fdca8097520 2023-07-02T09:01:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.455+0000 7fdcad3fd700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca8071270 con 0x7fdca8133880 2023-07-02T09:01:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca67fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdca8133880 0x7fdca8135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca6ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdca8097520 0x7fdca8097930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47446/0 (socket says 172.21.15.103:47446) 2023-07-02T09:01:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca6ffd700 1 -- 172.21.15.103:0/727711591 learned_addr learned my addr 172.21.15.103:0/727711591 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca67fc700 1 -- 172.21.15.103:0/727711591 >> v1:172.21.15.103:6789/0 conn(0x7fdca8097520 legacy=0x7fdca8097930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:44.459 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca67fc700 1 -- 172.21.15.103:0/727711591 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca809d1c0 con 0x7fdca8133880 2023-07-02T09:01:44.459 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca67fc700 1 --2- 172.21.15.103:0/727711591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8133880 0x7fdca8135ca0 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49302186c93eaff server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:44.459 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca5ffb700 1 -- 172.21.15.103:0/727711591 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc9c009cd0 con 0x7fdca8133880 2023-07-02T09:01:44.459 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca5ffb700 1 -- 172.21.15.103:0/727711591 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdc9c009e30 con 0x7fdca8133880 2023-07-02T09:01:44.459 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.456+0000 7fdca5ffb700 1 -- 172.21.15.103:0/727711591 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc9c00e360 con 0x7fdca8133880 2023-07-02T09:01:44.459 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.457+0000 7fdcad3fd700 1 -- 172.21.15.103:0/727711591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8133880 msgr2=0x7fdca8135ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:44.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.457+0000 7fdcad3fd700 1 --2- 172.21.15.103:0/727711591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8133880 0x7fdca8135ca0 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:44.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.457+0000 7fdcad3fd700 1 -- 172.21.15.103:0/727711591 shutdown_connections 2023-07-02T09:01:44.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.457+0000 7fdcad3fd700 1 --2- 172.21.15.103:0/727711591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8133880 0x7fdca8135ca0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:44.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.457+0000 7fdcad3fd700 1 -- 172.21.15.103:0/727711591 >> 172.21.15.103:0/727711591 conn(0x7fdca80a1030 msgr2=0x7fdca80a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:44.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.457+0000 7fdcad3fd700 1 -- 172.21.15.103:0/727711591 shutdown_connections 2023-07-02T09:01:44.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.457+0000 7fdcad3fd700 1 -- 172.21.15.103:0/727711591 wait complete. 2023-07-02T09:01:44.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.457+0000 7fdcad3fd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.458+0000 7fdcad3fd700 1 Processor -- start 2023-07-02T09:01:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.458+0000 7fdcad3fd700 1 -- start start 2023-07-02T09:01:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.459+0000 7fdcad3fd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8097520 0x7fdca8094380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.459+0000 7fdcad3fd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca8127a00 con 0x7fdca8097520 2023-07-02T09:01:44.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.459+0000 7fdca6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8097520 0x7fdca8094380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:44.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.459+0000 7fdca6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8097520 0x7fdca8094380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55992/0 (socket says 172.21.15.103:55992) 2023-07-02T09:01:44.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.459+0000 7fdca6ffd700 1 -- 172.21.15.103:0/1379139805 learned_addr learned my addr 172.21.15.103:0/1379139805 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:44.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.459+0000 7fdca6ffd700 1 -- 172.21.15.103:0/1379139805 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca8127530 con 0x7fdca8097520 2023-07-02T09:01:44.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.459+0000 7fdca6ffd700 1 --2- 172.21.15.103:0/1379139805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8097520 0x7fdca8094380 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:44.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.460+0000 7fdc8f7fe700 1 -- 172.21.15.103:0/1379139805 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc98005a60 con 0x7fdca8097520 2023-07-02T09:01:44.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.460+0000 7fdc8f7fe700 1 -- 172.21.15.103:0/1379139805 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdc98005bc0 con 0x7fdca8097520 2023-07-02T09:01:44.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.460+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca8128be0 con 0x7fdca8097520 2023-07-02T09:01:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.460+0000 7fdc8f7fe700 1 -- 172.21.15.103:0/1379139805 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc98005e60 con 0x7fdca8097520 2023-07-02T09:01:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.460+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca81295c0 con 0x7fdca8097520 2023-07-02T09:01:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.460+0000 7fdc8f7fe700 1 -- 172.21.15.103:0/1379139805 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fdc9800c7b0 con 0x7fdca8097520 2023-07-02T09:01:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.461+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca8127e80 con 0x7fdca8097520 2023-07-02T09:01:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.461+0000 7fdc8f7fe700 1 --2- 172.21.15.103:0/1379139805 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc940377c0 0x7fdc94039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.461+0000 7fdc8f7fe700 1 -- 172.21.15.103:0/1379139805 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fdc98046bf0 con 0x7fdca8097520 2023-07-02T09:01:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.461+0000 7fdca67fc700 1 --2- 172.21.15.103:0/1379139805 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc940377c0 0x7fdc94039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:44.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.461+0000 7fdca67fc700 1 --2- 172.21.15.103:0/1379139805 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc940377c0 0x7fdc94039cd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:44.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.465+0000 7fdc8f7fe700 1 -- 172.21.15.103:0/1379139805 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fdc9800a4a0 con 0x7fdca8097520 2023-07-02T09:01:44.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.655+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdca8061ea0 con 0x7fdca8097520 2023-07-02T09:01:44.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.655+0000 7fdc8f7fe700 1 -- 172.21.15.103:0/1379139805 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fdc9800a4a0 con 0x7fdca8097520 2023-07-02T09:01:44.657 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:44.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc940377c0 msgr2=0x7fdc94039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:44.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 --2- 172.21.15.103:0/1379139805 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc940377c0 0x7fdc94039cd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:44.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8097520 msgr2=0x7fdca8094380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:44.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 --2- 172.21.15.103:0/1379139805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8097520 0x7fdca8094380 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:44.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 shutdown_connections 2023-07-02T09:01:44.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 --2- 172.21.15.103:0/1379139805 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc940377c0 0x7fdc94039cd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:44.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 --2- 172.21.15.103:0/1379139805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdca8097520 0x7fdca8094380 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:44.660 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 >> 172.21.15.103:0/1379139805 conn(0x7fdca80a1030 msgr2=0x7fdca8095960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:44.660 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 shutdown_connections 2023-07-02T09:01:44.660 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:44.657+0000 7fdcad3fd700 1 -- 172.21.15.103:0/1379139805 wait complete. 2023-07-02T09:01:44.671 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:45.673 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:45.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.815+0000 7f4df8d43700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:45.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.819+0000 7f4df8d43700 1 Processor -- start 2023-07-02T09:01:45.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.819+0000 7f4df8d43700 1 -- start start 2023-07-02T09:01:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df8d43700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4df4132f50 0x7f4df4133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df8d43700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df409e260 con 0x7f4df409c580 2023-07-02T09:01:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df8d43700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df40a1640 con 0x7f4df4132f50 2023-07-02T09:01:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4df4132f50 0x7f4df4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df1d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4df409c580 0x7f4df409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47454/0 (socket says 172.21.15.103:47454) 2023-07-02T09:01:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df1d9b700 1 -- 172.21.15.103:0/3114448734 learned_addr learned my addr 172.21.15.103:0/3114448734 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4df4132f50 0x7f4df4133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55998/0 (socket says 172.21.15.103:55998) 2023-07-02T09:01:45.823 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df259c700 1 -- 172.21.15.103:0/3114448734 >> v1:172.21.15.103:6789/0 conn(0x7f4df409c580 legacy=0x7f4df409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:45.823 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df259c700 1 -- 172.21.15.103:0/3114448734 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df4132a20 con 0x7f4df4132f50 2023-07-02T09:01:45.823 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df159a700 1 -- 172.21.15.103:0/3114448734 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361773233 0 0) 0x7f4df409e260 con 0x7f4df409c580 2023-07-02T09:01:45.823 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df259c700 1 --2- 172.21.15.103:0/3114448734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df4132f50 0x7f4df4133360 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5452e9aad14b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:45.823 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df159a700 1 -- 172.21.15.103:0/3114448734 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4de8009cd0 con 0x7f4df4132f50 2023-07-02T09:01:45.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df159a700 1 -- 172.21.15.103:0/3114448734 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4de8009e30 con 0x7f4df4132f50 2023-07-02T09:01:45.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.820+0000 7f4df159a700 1 -- 172.21.15.103:0/3114448734 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4de800e360 con 0x7f4df4132f50 2023-07-02T09:01:45.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.821+0000 7f4df8d43700 1 -- 172.21.15.103:0/3114448734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df4132f50 msgr2=0x7f4df4133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:45.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.821+0000 7f4df8d43700 1 --2- 172.21.15.103:0/3114448734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df4132f50 0x7f4df4133360 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:45.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.821+0000 7f4df8d43700 1 -- 172.21.15.103:0/3114448734 shutdown_connections 2023-07-02T09:01:45.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.821+0000 7f4df8d43700 1 --2- 172.21.15.103:0/3114448734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df4132f50 0x7f4df4133360 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:45.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.821+0000 7f4df8d43700 1 -- 172.21.15.103:0/3114448734 >> 172.21.15.103:0/3114448734 conn(0x7f4df40a2080 msgr2=0x7f4df40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:45.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.821+0000 7f4df8d43700 1 -- 172.21.15.103:0/3114448734 shutdown_connections 2023-07-02T09:01:45.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.821+0000 7f4df8d43700 1 -- 172.21.15.103:0/3114448734 wait complete. 2023-07-02T09:01:45.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.821+0000 7f4df8d43700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:45.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.822+0000 7f4df8d43700 1 Processor -- start 2023-07-02T09:01:45.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.823+0000 7f4df8d43700 1 -- start start 2023-07-02T09:01:45.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.823+0000 7f4df8d43700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df409c580 0x7f4df41baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:45.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.823+0000 7f4df8d43700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df41ba610 con 0x7f4df409c580 2023-07-02T09:01:45.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.823+0000 7f4df259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df409c580 0x7f4df41baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:45.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.823+0000 7f4df259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df409c580 0x7f4df41baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56012/0 (socket says 172.21.15.103:56012) 2023-07-02T09:01:45.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.823+0000 7f4df259c700 1 -- 172.21.15.103:0/52179955 learned_addr learned my addr 172.21.15.103:0/52179955 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:45.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.824+0000 7f4df259c700 1 -- 172.21.15.103:0/52179955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df406dfc0 con 0x7f4df409c580 2023-07-02T09:01:45.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.824+0000 7f4df259c700 1 --2- 172.21.15.103:0/52179955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df409c580 0x7f4df41baa60 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:45.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.824+0000 7f4de2ffd700 1 -- 172.21.15.103:0/52179955 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4de8013530 con 0x7f4df409c580 2023-07-02T09:01:45.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.824+0000 7f4de2ffd700 1 -- 172.21.15.103:0/52179955 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4de8013690 con 0x7f4df409c580 2023-07-02T09:01:45.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.824+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df41ba270 con 0x7f4df409c580 2023-07-02T09:01:45.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.824+0000 7f4de2ffd700 1 -- 172.21.15.103:0/52179955 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4de8013930 con 0x7f4df409c580 2023-07-02T09:01:45.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.824+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df406dba0 con 0x7f4df409c580 2023-07-02T09:01:45.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.825+0000 7f4de2ffd700 1 -- 172.21.15.103:0/52179955 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4de8013ae0 con 0x7f4df409c580 2023-07-02T09:01:45.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.825+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df41ba460 con 0x7f4df409c580 2023-07-02T09:01:45.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.825+0000 7f4de2ffd700 1 --2- 172.21.15.103:0/52179955 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4dd40377b0 0x7f4dd4039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:45.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.825+0000 7f4de2ffd700 1 -- 172.21.15.103:0/52179955 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4de8018040 con 0x7f4df409c580 2023-07-02T09:01:45.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.826+0000 7f4df1d9b700 1 --2- 172.21.15.103:0/52179955 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4dd40377b0 0x7f4dd4039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:45.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.826+0000 7f4df1d9b700 1 --2- 172.21.15.103:0/52179955 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4dd40377b0 0x7f4dd4039cc0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:45.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:45.830+0000 7f4de2ffd700 1 -- 172.21.15.103:0/52179955 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f4de8018e00 con 0x7f4df409c580 2023-07-02T09:01:46.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.021+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4df4061ea0 con 0x7f4df409c580 2023-07-02T09:01:46.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.021+0000 7f4de2ffd700 1 -- 172.21.15.103:0/52179955 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4de8010e60 con 0x7f4df409c580 2023-07-02T09:01:46.023 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:46.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4dd40377b0 msgr2=0x7f4dd4039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:46.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 --2- 172.21.15.103:0/52179955 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4dd40377b0 0x7f4dd4039cc0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:46.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df409c580 msgr2=0x7f4df41baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:46.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 --2- 172.21.15.103:0/52179955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df409c580 0x7f4df41baa60 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:46.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 shutdown_connections 2023-07-02T09:01:46.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 --2- 172.21.15.103:0/52179955 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4dd40377b0 0x7f4dd4039cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:46.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 --2- 172.21.15.103:0/52179955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4df409c580 0x7f4df41baa60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:46.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 >> 172.21.15.103:0/52179955 conn(0x7f4df40a2080 msgr2=0x7f4df4130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:46.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 shutdown_connections 2023-07-02T09:01:46.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:46.024+0000 7f4df8d43700 1 -- 172.21.15.103:0/52179955 wait complete. 2023-07-02T09:01:46.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:46.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~121s 2023-07-02T09:01:47.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:47.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.178+0000 7f9c1d468700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:47.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.183+0000 7f9c1d468700 1 Processor -- start 2023-07-02T09:01:47.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.183+0000 7f9c1d468700 1 -- start start 2023-07-02T09:01:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c1d468700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9c18132f50 0x7f9c18133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c1d468700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1809e260 con 0x7f9c1809c580 2023-07-02T09:01:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c1d468700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c180a1640 con 0x7f9c18132f50 2023-07-02T09:01:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c16ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9c18132f50 0x7f9c18133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c16ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9c18132f50 0x7f9c18133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56022/0 (socket says 172.21.15.103:56022) 2023-07-02T09:01:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c167fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9c1809c580 0x7f9c1809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47470/0 (socket says 172.21.15.103:47470) 2023-07-02T09:01:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c16ffd700 1 -- 172.21.15.103:0/3569663083 learned_addr learned my addr 172.21.15.103:0/3569663083 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c16ffd700 1 -- 172.21.15.103:0/3569663083 >> v1:172.21.15.103:6789/0 conn(0x7f9c1809c580 legacy=0x7f9c1809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c16ffd700 1 -- 172.21.15.103:0/3569663083 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c18132a20 con 0x7f9c18132f50 2023-07-02T09:01:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.184+0000 7f9c15ffb700 1 -- 172.21.15.103:0/3569663083 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988716870 0 0) 0x7f9c1809e260 con 0x7f9c1809c580 2023-07-02T09:01:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c16ffd700 1 --2- 172.21.15.103:0/3569663083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c18132f50 0x7f9c18133360 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b1f72e34ee99ed server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c15ffb700 1 -- 172.21.15.103:0/3569663083 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9c0c009d90 con 0x7f9c18132f50 2023-07-02T09:01:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c15ffb700 1 -- 172.21.15.103:0/3569663083 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9c0c00e230 con 0x7f9c18132f50 2023-07-02T09:01:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c15ffb700 1 -- 172.21.15.103:0/3569663083 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9c0c00e4d0 con 0x7f9c18132f50 2023-07-02T09:01:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c1d468700 1 -- 172.21.15.103:0/3569663083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c18132f50 msgr2=0x7f9c18133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c1d468700 1 --2- 172.21.15.103:0/3569663083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c18132f50 0x7f9c18133360 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c1d468700 1 -- 172.21.15.103:0/3569663083 shutdown_connections 2023-07-02T09:01:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c1d468700 1 --2- 172.21.15.103:0/3569663083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c18132f50 0x7f9c18133360 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c1d468700 1 -- 172.21.15.103:0/3569663083 >> 172.21.15.103:0/3569663083 conn(0x7f9c180a2080 msgr2=0x7f9c180a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c1d468700 1 -- 172.21.15.103:0/3569663083 shutdown_connections 2023-07-02T09:01:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.185+0000 7f9c1d468700 1 -- 172.21.15.103:0/3569663083 wait complete. 2023-07-02T09:01:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.186+0000 7f9c1d468700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.187+0000 7f9c1d468700 1 Processor -- start 2023-07-02T09:01:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.187+0000 7f9c1d468700 1 -- start start 2023-07-02T09:01:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.187+0000 7f9c1d468700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c1809c580 0x7f9c181baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.187+0000 7f9c1d468700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c181ba610 con 0x7f9c1809c580 2023-07-02T09:01:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9c16ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c1809c580 0x7f9c181baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:47.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9c16ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c1809c580 0x7f9c181baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56024/0 (socket says 172.21.15.103:56024) 2023-07-02T09:01:47.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9c16ffd700 1 -- 172.21.15.103:0/3058658179 learned_addr learned my addr 172.21.15.103:0/3058658179 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:47.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9c16ffd700 1 -- 172.21.15.103:0/3058658179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1806dfc0 con 0x7f9c1809c580 2023-07-02T09:01:47.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9c16ffd700 1 --2- 172.21.15.103:0/3058658179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c1809c580 0x7f9c181baa60 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:47.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9bff7fe700 1 -- 172.21.15.103:0/3058658179 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9c0c014410 con 0x7f9c1809c580 2023-07-02T09:01:47.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9bff7fe700 1 -- 172.21.15.103:0/3058658179 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9c0c014570 con 0x7f9c1809c580 2023-07-02T09:01:47.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c181ba270 con 0x7f9c1809c580 2023-07-02T09:01:47.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9bff7fe700 1 -- 172.21.15.103:0/3058658179 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9c0c014810 con 0x7f9c1809c580 2023-07-02T09:01:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.188+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1806dba0 con 0x7f9c1809c580 2023-07-02T09:01:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.189+0000 7f9bff7fe700 1 -- 172.21.15.103:0/3058658179 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f9c0c0149c0 con 0x7f9c1809c580 2023-07-02T09:01:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.189+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c181ba460 con 0x7f9c1809c580 2023-07-02T09:01:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.189+0000 7f9bff7fe700 1 --2- 172.21.15.103:0/3058658179 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c040377c0 0x7f9c04039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.189+0000 7f9bff7fe700 1 -- 172.21.15.103:0/3058658179 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9c0c0482f0 con 0x7f9c1809c580 2023-07-02T09:01:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.190+0000 7f9c167fc700 1 --2- 172.21.15.103:0/3058658179 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c040377c0 0x7f9c04039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.190+0000 7f9c167fc700 1 --2- 172.21.15.103:0/3058658179 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c040377c0 0x7f9c04039cd0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:47.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.194+0000 7f9bff7fe700 1 -- 172.21.15.103:0/3058658179 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f9c0c00f8f0 con 0x7f9c1809c580 2023-07-02T09:01:47.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.383+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9c18061ea0 con 0x7f9c1809c580 2023-07-02T09:01:47.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.383+0000 7f9bff7fe700 1 -- 172.21.15.103:0/3058658179 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f9c0c00f710 con 0x7f9c1809c580 2023-07-02T09:01:47.384 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:47.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c040377c0 msgr2=0x7f9c04039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:47.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 --2- 172.21.15.103:0/3058658179 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c040377c0 0x7f9c04039cd0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:47.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c1809c580 msgr2=0x7f9c181baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:47.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 --2- 172.21.15.103:0/3058658179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c1809c580 0x7f9c181baa60 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:47.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 shutdown_connections 2023-07-02T09:01:47.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 --2- 172.21.15.103:0/3058658179 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9c040377c0 0x7f9c04039cd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:47.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 --2- 172.21.15.103:0/3058658179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c1809c580 0x7f9c181baa60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:47.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 >> 172.21.15.103:0/3058658179 conn(0x7f9c180a2080 msgr2=0x7f9c18130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:47.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 shutdown_connections 2023-07-02T09:01:47.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:47.385+0000 7f9c1d468700 1 -- 172.21.15.103:0/3058658179 wait complete. 2023-07-02T09:01:47.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:48.399 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:48.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.538+0000 7fb09a284700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:48.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.543+0000 7fb09a284700 1 Processor -- start 2023-07-02T09:01:48.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.543+0000 7fb09a284700 1 -- start start 2023-07-02T09:01:48.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.544+0000 7fb09a284700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb094132f50 0x7fb094133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.544+0000 7fb09a284700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09409e260 con 0x7fb09409c580 2023-07-02T09:01:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.544+0000 7fb09a284700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0940a1640 con 0x7fb094132f50 2023-07-02T09:01:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.544+0000 7fb093fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb094132f50 0x7fb094133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.544+0000 7fb0937fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb09409c580 0x7fb09409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47484/0 (socket says 172.21.15.103:47484) 2023-07-02T09:01:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.544+0000 7fb093fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb094132f50 0x7fb094133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56026/0 (socket says 172.21.15.103:56026) 2023-07-02T09:01:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.544+0000 7fb093fff700 1 -- 172.21.15.103:0/1081486447 learned_addr learned my addr 172.21.15.103:0/1081486447 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:48.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.545+0000 7fb093fff700 1 -- 172.21.15.103:0/1081486447 >> v1:172.21.15.103:6789/0 conn(0x7fb09409c580 legacy=0x7fb09409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:48.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.545+0000 7fb093fff700 1 -- 172.21.15.103:0/1081486447 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094132a20 con 0x7fb094132f50 2023-07-02T09:01:48.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.545+0000 7fb093fff700 1 --2- 172.21.15.103:0/1081486447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb094132f50 0x7fb094133360 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cd24247ece7755 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:48.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.545+0000 7fb092ffd700 1 -- 172.21.15.103:0/1081486447 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb07c009d90 con 0x7fb094132f50 2023-07-02T09:01:48.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.545+0000 7fb092ffd700 1 -- 172.21.15.103:0/1081486447 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb07c00e230 con 0x7fb094132f50 2023-07-02T09:01:48.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.545+0000 7fb092ffd700 1 -- 172.21.15.103:0/1081486447 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb07c00e4d0 con 0x7fb094132f50 2023-07-02T09:01:48.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.546+0000 7fb09a284700 1 -- 172.21.15.103:0/1081486447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb094132f50 msgr2=0x7fb094133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.546+0000 7fb09a284700 1 --2- 172.21.15.103:0/1081486447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb094132f50 0x7fb094133360 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.546+0000 7fb09a284700 1 -- 172.21.15.103:0/1081486447 shutdown_connections 2023-07-02T09:01:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.546+0000 7fb09a284700 1 --2- 172.21.15.103:0/1081486447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb094132f50 0x7fb094133360 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.546+0000 7fb09a284700 1 -- 172.21.15.103:0/1081486447 >> 172.21.15.103:0/1081486447 conn(0x7fb0940a2080 msgr2=0x7fb0940a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.546+0000 7fb09a284700 1 -- 172.21.15.103:0/1081486447 shutdown_connections 2023-07-02T09:01:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.546+0000 7fb09a284700 1 -- 172.21.15.103:0/1081486447 wait complete. 2023-07-02T09:01:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.546+0000 7fb09a284700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.547+0000 7fb09a284700 1 Processor -- start 2023-07-02T09:01:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.547+0000 7fb09a284700 1 -- start start 2023-07-02T09:01:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb09a284700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09409c580 0x7fb09412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb09a284700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09406f9d0 con 0x7fb09409c580 2023-07-02T09:01:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb093fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09409c580 0x7fb09412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:48.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb093fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09409c580 0x7fb09412b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56040/0 (socket says 172.21.15.103:56040) 2023-07-02T09:01:48.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb093fff700 1 -- 172.21.15.103:0/3128819510 learned_addr learned my addr 172.21.15.103:0/3128819510 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:48.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb093fff700 1 -- 172.21.15.103:0/3128819510 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094133a00 con 0x7fb09409c580 2023-07-02T09:01:48.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb093fff700 1 --2- 172.21.15.103:0/3128819510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09409c580 0x7fb09412b7b0 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:48.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb090ff9700 1 -- 172.21.15.103:0/3128819510 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb07c014540 con 0x7fb09409c580 2023-07-02T09:01:48.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb090ff9700 1 -- 172.21.15.103:0/3128819510 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb07c0146a0 con 0x7fb09409c580 2023-07-02T09:01:48.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb090ff9700 1 -- 172.21.15.103:0/3128819510 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb07c014940 con 0x7fb09409c580 2023-07-02T09:01:48.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09406d160 con 0x7fb09409c580 2023-07-02T09:01:48.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.548+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09406da70 con 0x7fb09409c580 2023-07-02T09:01:48.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.549+0000 7fb090ff9700 1 -- 172.21.15.103:0/3128819510 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb07c014af0 con 0x7fb09409c580 2023-07-02T09:01:48.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.549+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094061ea0 con 0x7fb09409c580 2023-07-02T09:01:48.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.549+0000 7fb090ff9700 1 --2- 172.21.15.103:0/3128819510 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb0740377b0 0x7fb074039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:48.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.549+0000 7fb090ff9700 1 -- 172.21.15.103:0/3128819510 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb07c048350 con 0x7fb09409c580 2023-07-02T09:01:48.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.550+0000 7fb0937fe700 1 --2- 172.21.15.103:0/3128819510 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb0740377b0 0x7fb074039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:48.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.550+0000 7fb0937fe700 1 --2- 172.21.15.103:0/3128819510 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb0740377b0 0x7fb074039cc0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:48.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.554+0000 7fb090ff9700 1 -- 172.21.15.103:0/3128819510 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb07c00fe40 con 0x7fb09409c580 2023-07-02T09:01:48.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.745+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb0940725b0 con 0x7fb09409c580 2023-07-02T09:01:48.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.746+0000 7fb090ff9700 1 -- 172.21.15.103:0/3128819510 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb07c010e60 con 0x7fb09409c580 2023-07-02T09:01:48.746 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:48.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb0740377b0 msgr2=0x7fb074039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:48.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 --2- 172.21.15.103:0/3128819510 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb0740377b0 0x7fb074039cc0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:48.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09409c580 msgr2=0x7fb09412b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:48.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 --2- 172.21.15.103:0/3128819510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09409c580 0x7fb09412b7b0 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:48.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 shutdown_connections 2023-07-02T09:01:48.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 --2- 172.21.15.103:0/3128819510 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb0740377b0 0x7fb074039cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:48.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 --2- 172.21.15.103:0/3128819510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb09409c580 0x7fb09412b7b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:48.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 >> 172.21.15.103:0/3128819510 conn(0x7fb0940a2080 msgr2=0x7fb0941babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:48.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 shutdown_connections 2023-07-02T09:01:48.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:48.748+0000 7fb09a284700 1 -- 172.21.15.103:0/3128819510 wait complete. 2023-07-02T09:01:48.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:49.762 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:49.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.901+0000 7f02e8806700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:49.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.906+0000 7f02e8806700 1 Processor -- start 2023-07-02T09:01:49.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.906+0000 7f02e8806700 1 -- start start 2023-07-02T09:01:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e8806700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f02e4133360 0x7f02e409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e8806700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e409e260 con 0x7f02e4132f50 2023-07-02T09:01:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e8806700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e40a1640 con 0x7f02e4133360 2023-07-02T09:01:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e1d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f02e4133360 0x7f02e409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e259c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f02e4132f50 0x7f02e409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47488/0 (socket says 172.21.15.103:47488) 2023-07-02T09:01:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e259c700 1 -- 172.21.15.103:0/3439268013 learned_addr learned my addr 172.21.15.103:0/3439268013 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e1d9b700 1 -- 172.21.15.103:0/3439268013 >> v1:172.21.15.103:6789/0 conn(0x7f02e4132f50 legacy=0x7f02e409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e1d9b700 1 -- 172.21.15.103:0/3439268013 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4132a20 con 0x7f02e4133360 2023-07-02T09:01:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e159a700 1 -- 172.21.15.103:0/3439268013 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402519288 0 0) 0x7f02e409e260 con 0x7f02e4132f50 2023-07-02T09:01:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.907+0000 7f02e1d9b700 1 --2- 172.21.15.103:0/3439268013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4133360 0x7f02e409cc90 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e92c67171fbdf82 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e159a700 1 -- 172.21.15.103:0/3439268013 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f02d4009cd0 con 0x7f02e4133360 2023-07-02T09:01:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e159a700 1 -- 172.21.15.103:0/3439268013 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f02d4009e30 con 0x7f02e4133360 2023-07-02T09:01:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e159a700 1 -- 172.21.15.103:0/3439268013 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f02d400e360 con 0x7f02e4133360 2023-07-02T09:01:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e8806700 1 -- 172.21.15.103:0/3439268013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4133360 msgr2=0x7f02e409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e8806700 1 --2- 172.21.15.103:0/3439268013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4133360 0x7f02e409cc90 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e8806700 1 -- 172.21.15.103:0/3439268013 shutdown_connections 2023-07-02T09:01:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e8806700 1 --2- 172.21.15.103:0/3439268013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4133360 0x7f02e409cc90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e8806700 1 -- 172.21.15.103:0/3439268013 >> 172.21.15.103:0/3439268013 conn(0x7f02e40a2080 msgr2=0x7f02e40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e8806700 1 -- 172.21.15.103:0/3439268013 shutdown_connections 2023-07-02T09:01:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.908+0000 7f02e8806700 1 -- 172.21.15.103:0/3439268013 wait complete. 2023-07-02T09:01:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.909+0000 7f02e8806700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:49.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.910+0000 7f02e8806700 1 Processor -- start 2023-07-02T09:01:49.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.910+0000 7f02e8806700 1 -- start start 2023-07-02T09:01:49.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.910+0000 7f02e8806700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4132f50 0x7f02e41ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:49.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.910+0000 7f02e8806700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e406d370 con 0x7f02e4132f50 2023-07-02T09:01:49.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.910+0000 7f02e259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4132f50 0x7f02e41ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:49.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02e259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4132f50 0x7f02e41ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56064/0 (socket says 172.21.15.103:56064) 2023-07-02T09:01:49.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02e259c700 1 -- 172.21.15.103:0/2170403449 learned_addr learned my addr 172.21.15.103:0/2170403449 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:49.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02e259c700 1 -- 172.21.15.103:0/2170403449 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e406dde0 con 0x7f02e4132f50 2023-07-02T09:01:49.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02e259c700 1 --2- 172.21.15.103:0/2170403449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4132f50 0x7f02e41ba9b0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:49.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02d2ffd700 1 -- 172.21.15.103:0/2170403449 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f02d8004b80 con 0x7f02e4132f50 2023-07-02T09:01:49.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02d2ffd700 1 -- 172.21.15.103:0/2170403449 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f02d8004ce0 con 0x7f02e4132f50 2023-07-02T09:01:49.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e41ba4f0 con 0x7f02e4132f50 2023-07-02T09:01:49.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02d2ffd700 1 -- 172.21.15.103:0/2170403449 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f02d800c4f0 con 0x7f02e4132f50 2023-07-02T09:01:49.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.911+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e4071e40 con 0x7f02e4132f50 2023-07-02T09:01:49.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.912+0000 7f02d2ffd700 1 -- 172.21.15.103:0/2170403449 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f02d800c650 con 0x7f02e4132f50 2023-07-02T09:01:49.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.912+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e4061ea0 con 0x7f02e4132f50 2023-07-02T09:01:49.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.912+0000 7f02d2ffd700 1 --2- 172.21.15.103:0/2170403449 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f02c40377c0 0x7f02c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:49.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.912+0000 7f02d2ffd700 1 -- 172.21.15.103:0/2170403449 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f02d8046970 con 0x7f02e4132f50 2023-07-02T09:01:49.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.913+0000 7f02e1d9b700 1 --2- 172.21.15.103:0/2170403449 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f02c40377c0 0x7f02c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:49.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.913+0000 7f02e1d9b700 1 --2- 172.21.15.103:0/2170403449 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f02c40377c0 0x7f02c4039cd0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:49.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:49.917+0000 7f02d2ffd700 1 -- 172.21.15.103:0/2170403449 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f02d8009c80 con 0x7f02e4132f50 2023-07-02T09:01:50.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.108+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f02e40725b0 con 0x7f02e4132f50 2023-07-02T09:01:50.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.108+0000 7f02d2ffd700 1 -- 172.21.15.103:0/2170403449 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f02d80098a0 con 0x7f02e4132f50 2023-07-02T09:01:50.109 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:50.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.110+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f02c40377c0 msgr2=0x7f02c4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:50.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.110+0000 7f02e8806700 1 --2- 172.21.15.103:0/2170403449 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f02c40377c0 0x7f02c4039cd0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:50.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.111+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4132f50 msgr2=0x7f02e41ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:50.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.111+0000 7f02e8806700 1 --2- 172.21.15.103:0/2170403449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4132f50 0x7f02e41ba9b0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:50.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.111+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 shutdown_connections 2023-07-02T09:01:50.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.111+0000 7f02e8806700 1 --2- 172.21.15.103:0/2170403449 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f02c40377c0 0x7f02c4039cd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:50.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.111+0000 7f02e8806700 1 --2- 172.21.15.103:0/2170403449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02e4132f50 0x7f02e41ba9b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:50.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.111+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 >> 172.21.15.103:0/2170403449 conn(0x7f02e40a2080 msgr2=0x7f02e40a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:50.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.111+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 shutdown_connections 2023-07-02T09:01:50.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:50.111+0000 7f02e8806700 1 -- 172.21.15.103:0/2170403449 wait complete. 2023-07-02T09:01:50.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:51.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:51.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.261+0000 7f400790e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.266+0000 7f400790e700 1 Processor -- start 2023-07-02T09:01:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.266+0000 7f400790e700 1 -- start start 2023-07-02T09:01:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f400790e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4000133360 0x7f400009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f400790e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400009e260 con 0x7f4000132f50 2023-07-02T09:01:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f400790e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40000a1640 con 0x7f4000133360 2023-07-02T09:01:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f4004ea9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4000133360 0x7f400009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f40056aa700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4000132f50 0x7f400009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47492/0 (socket says 172.21.15.103:47492) 2023-07-02T09:01:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f4004ea9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4000133360 0x7f400009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56068/0 (socket says 172.21.15.103:56068) 2023-07-02T09:01:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f40056aa700 1 -- 172.21.15.103:0/3938475529 learned_addr learned my addr 172.21.15.103:0/3938475529 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f4004ea9700 1 -- 172.21.15.103:0/3938475529 >> v1:172.21.15.103:6789/0 conn(0x7f4000132f50 legacy=0x7f400009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f4004ea9700 1 -- 172.21.15.103:0/3938475529 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4000132a20 con 0x7f4000133360 2023-07-02T09:01:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f4004ea9700 1 --2- 172.21.15.103:0/3938475529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000133360 0x7f400009cc90 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d1b8d0563481f7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f3ff7fff700 1 -- 172.21.15.103:0/3938475529 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3ff0009cd0 con 0x7f4000133360 2023-07-02T09:01:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f3ff7fff700 1 -- 172.21.15.103:0/3938475529 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3ff0009e30 con 0x7f4000133360 2023-07-02T09:01:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.267+0000 7f3ff7fff700 1 -- 172.21.15.103:0/3938475529 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3ff000e360 con 0x7f4000133360 2023-07-02T09:01:51.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.268+0000 7f400790e700 1 -- 172.21.15.103:0/3938475529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000133360 msgr2=0x7f400009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:51.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.268+0000 7f400790e700 1 --2- 172.21.15.103:0/3938475529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000133360 0x7f400009cc90 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:51.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.268+0000 7f400790e700 1 -- 172.21.15.103:0/3938475529 shutdown_connections 2023-07-02T09:01:51.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.268+0000 7f400790e700 1 --2- 172.21.15.103:0/3938475529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000133360 0x7f400009cc90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:51.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.268+0000 7f400790e700 1 -- 172.21.15.103:0/3938475529 >> 172.21.15.103:0/3938475529 conn(0x7f40000a2080 msgr2=0x7f40000a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:51.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.268+0000 7f400790e700 1 -- 172.21.15.103:0/3938475529 shutdown_connections 2023-07-02T09:01:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.268+0000 7f400790e700 1 -- 172.21.15.103:0/3938475529 wait complete. 2023-07-02T09:01:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.268+0000 7f400790e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.269+0000 7f400790e700 1 Processor -- start 2023-07-02T09:01:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.269+0000 7f400790e700 1 -- start start 2023-07-02T09:01:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.270+0000 7f400790e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000132f50 0x7f400012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.270+0000 7f400790e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400006d260 con 0x7f4000132f50 2023-07-02T09:01:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.270+0000 7f40056aa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000132f50 0x7f400012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.270+0000 7f40056aa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000132f50 0x7f400012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56072/0 (socket says 172.21.15.103:56072) 2023-07-02T09:01:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.270+0000 7f40056aa700 1 -- 172.21.15.103:0/449367836 learned_addr learned my addr 172.21.15.103:0/449367836 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.270+0000 7f40056aa700 1 -- 172.21.15.103:0/449367836 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400012bb90 con 0x7f4000132f50 2023-07-02T09:01:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.270+0000 7f40056aa700 1 --2- 172.21.15.103:0/449367836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000132f50 0x7f400012bea0 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.270+0000 7f3ff5ffb700 1 -- 172.21.15.103:0/449367836 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3ffc005a60 con 0x7f4000132f50 2023-07-02T09:01:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.271+0000 7f3ff5ffb700 1 -- 172.21.15.103:0/449367836 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3ffc005bc0 con 0x7f4000132f50 2023-07-02T09:01:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.271+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400006f730 con 0x7f4000132f50 2023-07-02T09:01:51.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.271+0000 7f3ff5ffb700 1 -- 172.21.15.103:0/449367836 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3ffc005e60 con 0x7f4000132f50 2023-07-02T09:01:51.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.271+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400006f490 con 0x7f4000132f50 2023-07-02T09:01:51.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.271+0000 7f3ff5ffb700 1 -- 172.21.15.103:0/449367836 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3ffc0070b0 con 0x7f4000132f50 2023-07-02T09:01:51.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.272+0000 7f3ff5ffb700 1 --2- 172.21.15.103:0/449367836 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fe40377c0 0x7f3fe4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:51.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.272+0000 7f3ff5ffb700 1 -- 172.21.15.103:0/449367836 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3ffc0469e0 con 0x7f4000132f50 2023-07-02T09:01:51.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.272+0000 7f4004ea9700 1 --2- 172.21.15.103:0/449367836 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fe40377c0 0x7f3fe4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:51.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.272+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400006f490 con 0x7f4000132f50 2023-07-02T09:01:51.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.272+0000 7f4004ea9700 1 --2- 172.21.15.103:0/449367836 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fe40377c0 0x7f3fe4039cd0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:51.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.277+0000 7f3ff5ffb700 1 -- 172.21.15.103:0/449367836 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f400006f490 con 0x7f4000132f50 2023-07-02T09:01:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.466+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f400006f490 con 0x7f4000132f50 2023-07-02T09:01:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.466+0000 7f3ff5ffb700 1 -- 172.21.15.103:0/449367836 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f400006f490 con 0x7f4000132f50 2023-07-02T09:01:51.467 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:51.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fe40377c0 msgr2=0x7f3fe4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:51.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 --2- 172.21.15.103:0/449367836 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fe40377c0 0x7f3fe4039cd0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:51.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000132f50 msgr2=0x7f400012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:51.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 --2- 172.21.15.103:0/449367836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000132f50 0x7f400012bea0 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:51.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 shutdown_connections 2023-07-02T09:01:51.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 --2- 172.21.15.103:0/449367836 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3fe40377c0 0x7f3fe4039cd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:51.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 --2- 172.21.15.103:0/449367836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4000132f50 0x7f400012bea0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:51.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 >> 172.21.15.103:0/449367836 conn(0x7f40000a2080 msgr2=0x7f40001bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:51.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 shutdown_connections 2023-07-02T09:01:51.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:51.468+0000 7f400790e700 1 -- 172.21.15.103:0/449367836 wait complete. 2023-07-02T09:01:51.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:52.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~127s 2023-07-02T09:01:52.482 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:52.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.618+0000 7f1f7fba0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:52.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.623+0000 7f1f7fba0700 1 Processor -- start 2023-07-02T09:01:52.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.624+0000 7f1f7fba0700 1 -- start start 2023-07-02T09:01:52.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.624+0000 7f1f7fba0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1f78133360 0x7f1f7809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:52.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.624+0000 7f1f7fba0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7809e260 con 0x7f1f78132f50 2023-07-02T09:01:52.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.624+0000 7f1f7fba0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780a1640 con 0x7f1f78133360 2023-07-02T09:01:52.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.624+0000 7f1f7d93c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1f78132f50 0x7f1f7809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:47498/0 (socket says 172.21.15.103:47498) 2023-07-02T09:01:52.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.624+0000 7f1f7d93c700 1 -- 172.21.15.103:0/215640605 learned_addr learned my addr 172.21.15.103:0/215640605 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:52.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.624+0000 7f1f7d13b700 1 --2- 172.21.15.103:0/215640605 >> v2:172.21.15.103:3300/0 conn(0x7f1f78133360 0x7f1f7809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:52.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.625+0000 7f1f7c93a700 1 -- 172.21.15.103:0/215640605 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131470632 0 0) 0x7f1f7809e260 con 0x7f1f78132f50 2023-07-02T09:01:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.625+0000 7f1f7c93a700 1 -- 172.21.15.103:0/215640605 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f64003580 con 0x7f1f78132f50 2023-07-02T09:01:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.625+0000 7f1f7d13b700 1 -- 172.21.15.103:0/215640605 >> v1:172.21.15.103:6789/0 conn(0x7f1f78132f50 legacy=0x7f1f7809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.625+0000 7f1f7d13b700 1 -- 172.21.15.103:0/215640605 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78132a20 con 0x7f1f78133360 2023-07-02T09:01:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.625+0000 7f1f7d13b700 1 --2- 172.21.15.103:0/215640605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78133360 0x7f1f7809cc90 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c228237ea9c755b6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.625+0000 7f1f7c93a700 1 -- 172.21.15.103:0/215640605 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1f74009c80 con 0x7f1f78133360 2023-07-02T09:01:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.625+0000 7f1f7c93a700 1 -- 172.21.15.103:0/215640605 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1f74009de0 con 0x7f1f78133360 2023-07-02T09:01:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.625+0000 7f1f7c93a700 1 -- 172.21.15.103:0/215640605 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1f7400e360 con 0x7f1f78133360 2023-07-02T09:01:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.626+0000 7f1f7fba0700 1 -- 172.21.15.103:0/215640605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78133360 msgr2=0x7f1f7809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.626+0000 7f1f7fba0700 1 --2- 172.21.15.103:0/215640605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78133360 0x7f1f7809cc90 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.626+0000 7f1f7fba0700 1 -- 172.21.15.103:0/215640605 shutdown_connections 2023-07-02T09:01:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.626+0000 7f1f7fba0700 1 --2- 172.21.15.103:0/215640605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78133360 0x7f1f7809cc90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.626+0000 7f1f7fba0700 1 -- 172.21.15.103:0/215640605 >> 172.21.15.103:0/215640605 conn(0x7f1f780a2080 msgr2=0x7f1f780a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.626+0000 7f1f7fba0700 1 -- 172.21.15.103:0/215640605 shutdown_connections 2023-07-02T09:01:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.626+0000 7f1f7fba0700 1 -- 172.21.15.103:0/215640605 wait complete. 2023-07-02T09:01:52.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.626+0000 7f1f7fba0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:52.630 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.627+0000 7f1f7fba0700 1 Processor -- start 2023-07-02T09:01:52.630 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.627+0000 7f1f7fba0700 1 -- start start 2023-07-02T09:01:52.630 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7fba0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78132f50 0x7f1f7812c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7fba0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7812beb0 con 0x7f1f78132f50 2023-07-02T09:01:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7d93c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78132f50 0x7f1f7812c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7d93c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78132f50 0x7f1f7812c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56092/0 (socket says 172.21.15.103:56092) 2023-07-02T09:01:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7d93c700 1 -- 172.21.15.103:0/2712923990 learned_addr learned my addr 172.21.15.103:0/2712923990 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7d93c700 1 -- 172.21.15.103:0/2712923990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7806fb30 con 0x7f1f78132f50 2023-07-02T09:01:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7d93c700 1 --2- 172.21.15.103:0/2712923990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78132f50 0x7f1f7812c270 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f6e7fc700 1 -- 172.21.15.103:0/2712923990 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1f68005a60 con 0x7f1f78132f50 2023-07-02T09:01:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f6e7fc700 1 -- 172.21.15.103:0/2712923990 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1f68005bc0 con 0x7f1f78132f50 2023-07-02T09:01:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7812bb10 con 0x7f1f78132f50 2023-07-02T09:01:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f6e7fc700 1 -- 172.21.15.103:0/2712923990 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1f68005e60 con 0x7f1f78132f50 2023-07-02T09:01:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.628+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7806f5f0 con 0x7f1f78132f50 2023-07-02T09:01:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.629+0000 7f1f6e7fc700 1 -- 172.21.15.103:0/2712923990 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f1f680070b0 con 0x7f1f78132f50 2023-07-02T09:01:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.629+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7812bd00 con 0x7f1f78132f50 2023-07-02T09:01:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.629+0000 7f1f6e7fc700 1 --2- 172.21.15.103:0/2712923990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1f5c0377c0 0x7f1f5c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.630+0000 7f1f6e7fc700 1 -- 172.21.15.103:0/2712923990 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f1f68046b50 con 0x7f1f78132f50 2023-07-02T09:01:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.630+0000 7f1f7d13b700 1 --2- 172.21.15.103:0/2712923990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1f5c0377c0 0x7f1f5c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.630+0000 7f1f7d13b700 1 --2- 172.21.15.103:0/2712923990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1f5c0377c0 0x7f1f5c039cd0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:52.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.634+0000 7f1f6e7fc700 1 -- 172.21.15.103:0/2712923990 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f1f6800a4e0 con 0x7f1f78132f50 2023-07-02T09:01:52.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.827+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1f780a1920 con 0x7f1f78132f50 2023-07-02T09:01:52.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.827+0000 7f1f6e7fc700 1 -- 172.21.15.103:0/2712923990 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f1f6800a120 con 0x7f1f78132f50 2023-07-02T09:01:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:52.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1f5c0377c0 msgr2=0x7f1f5c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:52.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 --2- 172.21.15.103:0/2712923990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1f5c0377c0 0x7f1f5c039cd0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:52.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78132f50 msgr2=0x7f1f7812c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:52.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 --2- 172.21.15.103:0/2712923990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78132f50 0x7f1f7812c270 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:52.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 shutdown_connections 2023-07-02T09:01:52.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 --2- 172.21.15.103:0/2712923990 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1f5c0377c0 0x7f1f5c039cd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:52.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 --2- 172.21.15.103:0/2712923990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f78132f50 0x7f1f7812c270 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:52.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 >> 172.21.15.103:0/2712923990 conn(0x7f1f780a2080 msgr2=0x7f1f781bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:52.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 shutdown_connections 2023-07-02T09:01:52.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:52.829+0000 7f1f7fba0700 1 -- 172.21.15.103:0/2712923990 wait complete. 2023-07-02T09:01:52.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:53.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:53.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.982+0000 7f8bdd844700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:53.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.987+0000 7f8bdd844700 1 Processor -- start 2023-07-02T09:01:53.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.987+0000 7f8bdd844700 1 -- start start 2023-07-02T09:01:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.988+0000 7f8bdd844700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8bd8132f50 0x7f8bd8133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.988+0000 7f8bdd844700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd809e260 con 0x7f8bd809c580 2023-07-02T09:01:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.988+0000 7f8bdd844700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd80a1640 con 0x7f8bd8132f50 2023-07-02T09:01:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.988+0000 7f8bd6ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8bd8132f50 0x7f8bd8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.988+0000 7f8bd67fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f8bd809c580 0x7f8bd809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50312/0 (socket says 172.21.15.103:50312) 2023-07-02T09:01:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.988+0000 7f8bd6ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8bd8132f50 0x7f8bd8133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60366/0 (socket says 172.21.15.103:60366) 2023-07-02T09:01:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.988+0000 7f8bd67fc700 1 -- 172.21.15.103:0/531724250 learned_addr learned my addr 172.21.15.103:0/531724250 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.988+0000 7f8bd6ffd700 1 -- 172.21.15.103:0/531724250 >> v1:172.21.15.103:6789/0 conn(0x7f8bd809c580 legacy=0x7f8bd809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.989+0000 7f8bd6ffd700 1 -- 172.21.15.103:0/531724250 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd8132a20 con 0x7f8bd8132f50 2023-07-02T09:01:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.989+0000 7f8bd5ffb700 1 -- 172.21.15.103:0/531724250 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565770324 0 0) 0x7f8bd809e260 con 0x7f8bd809c580 2023-07-02T09:01:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.989+0000 7f8bd6ffd700 1 --2- 172.21.15.103:0/531724250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd8132f50 0x7f8bd8133360 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2591ccfa78d054 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.989+0000 7f8bd5ffb700 1 -- 172.21.15.103:0/531724250 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8bcc009cd0 con 0x7f8bd8132f50 2023-07-02T09:01:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.989+0000 7f8bd5ffb700 1 -- 172.21.15.103:0/531724250 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8bcc009e30 con 0x7f8bd8132f50 2023-07-02T09:01:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.989+0000 7f8bd5ffb700 1 -- 172.21.15.103:0/531724250 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8bcc00e360 con 0x7f8bd8132f50 2023-07-02T09:01:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.989+0000 7f8bdd844700 1 -- 172.21.15.103:0/531724250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd8132f50 msgr2=0x7f8bd8133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.989+0000 7f8bdd844700 1 --2- 172.21.15.103:0/531724250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd8132f50 0x7f8bd8133360 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.990+0000 7f8bdd844700 1 -- 172.21.15.103:0/531724250 shutdown_connections 2023-07-02T09:01:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.990+0000 7f8bdd844700 1 --2- 172.21.15.103:0/531724250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd8132f50 0x7f8bd8133360 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.990+0000 7f8bdd844700 1 -- 172.21.15.103:0/531724250 >> 172.21.15.103:0/531724250 conn(0x7f8bd80a2080 msgr2=0x7f8bd80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.990+0000 7f8bdd844700 1 -- 172.21.15.103:0/531724250 shutdown_connections 2023-07-02T09:01:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.990+0000 7f8bdd844700 1 -- 172.21.15.103:0/531724250 wait complete. 2023-07-02T09:01:53.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.990+0000 7f8bdd844700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:53.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.991+0000 7f8bdd844700 1 Processor -- start 2023-07-02T09:01:53.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.991+0000 7f8bdd844700 1 -- start start 2023-07-02T09:01:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.992+0000 7f8bdd844700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd809c580 0x7f8bd812beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.992+0000 7f8bdd844700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd8133a00 con 0x7f8bd809c580 2023-07-02T09:01:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.992+0000 7f8bd6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd809c580 0x7f8bd812beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.992+0000 7f8bd6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd809c580 0x7f8bd812beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60372/0 (socket says 172.21.15.103:60372) 2023-07-02T09:01:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.992+0000 7f8bd6ffd700 1 -- 172.21.15.103:0/3712895630 learned_addr learned my addr 172.21.15.103:0/3712895630 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.993+0000 7f8bd6ffd700 1 -- 172.21.15.103:0/3712895630 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd806dc00 con 0x7f8bd809c580 2023-07-02T09:01:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.993+0000 7f8bd6ffd700 1 --2- 172.21.15.103:0/3712895630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd809c580 0x7f8bd812beb0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.993+0000 7f8bbffff700 1 -- 172.21.15.103:0/3712895630 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8bcc014540 con 0x7f8bd809c580 2023-07-02T09:01:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.993+0000 7f8bbffff700 1 -- 172.21.15.103:0/3712895630 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8bcc0146a0 con 0x7f8bd809c580 2023-07-02T09:01:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.993+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd806e920 con 0x7f8bd809c580 2023-07-02T09:01:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.993+0000 7f8bbffff700 1 -- 172.21.15.103:0/3712895630 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8bcc014940 con 0x7f8bd809c580 2023-07-02T09:01:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.993+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd8071f80 con 0x7f8bd809c580 2023-07-02T09:01:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.994+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd8061ea0 con 0x7f8bd809c580 2023-07-02T09:01:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.994+0000 7f8bbffff700 1 -- 172.21.15.103:0/3712895630 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f8bcc014af0 con 0x7f8bd809c580 2023-07-02T09:01:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.994+0000 7f8bbffff700 1 --2- 172.21.15.103:0/3712895630 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8bc40377c0 0x7f8bc4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.994+0000 7f8bbffff700 1 -- 172.21.15.103:0/3712895630 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f8bcc048340 con 0x7f8bd809c580 2023-07-02T09:01:53.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.994+0000 7f8bd67fc700 1 --2- 172.21.15.103:0/3712895630 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8bc40377c0 0x7f8bc4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:53.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.995+0000 7f8bd67fc700 1 --2- 172.21.15.103:0/3712895630 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8bc40377c0 0x7f8bc4039cd0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:53.999 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:53.999+0000 7f8bbffff700 1 -- 172.21.15.103:0/3712895630 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f8bcc00fc20 con 0x7f8bd809c580 2023-07-02T09:01:54.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.188+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8bd80725b0 con 0x7f8bd809c580 2023-07-02T09:01:54.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.189+0000 7f8bbffff700 1 -- 172.21.15.103:0/3712895630 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f8bcc011320 con 0x7f8bd809c580 2023-07-02T09:01:54.189 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:54.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8bc40377c0 msgr2=0x7f8bc4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:54.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 --2- 172.21.15.103:0/3712895630 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8bc40377c0 0x7f8bc4039cd0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:54.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd809c580 msgr2=0x7f8bd812beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:54.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 --2- 172.21.15.103:0/3712895630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd809c580 0x7f8bd812beb0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:54.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 shutdown_connections 2023-07-02T09:01:54.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 --2- 172.21.15.103:0/3712895630 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f8bc40377c0 0x7f8bc4039cd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:54.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 --2- 172.21.15.103:0/3712895630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd809c580 0x7f8bd812beb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:54.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 >> 172.21.15.103:0/3712895630 conn(0x7f8bd80a2080 msgr2=0x7f8bd81ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:54.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 shutdown_connections 2023-07-02T09:01:54.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:54.191+0000 7f8bdd844700 1 -- 172.21.15.103:0/3712895630 wait complete. 2023-07-02T09:01:54.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:55.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:55.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.343+0000 7fd8e297f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:55.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.348+0000 7fd8e297f700 1 Processor -- start 2023-07-02T09:01:55.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.348+0000 7fd8e297f700 1 -- start start 2023-07-02T09:01:55.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.349+0000 7fd8e297f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd8dc133360 0x7fd8dc09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:55.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.349+0000 7fd8e297f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc09e260 con 0x7fd8dc132f50 2023-07-02T09:01:55.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.349+0000 7fd8e297f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8dc0a1640 con 0x7fd8dc133360 2023-07-02T09:01:55.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.349+0000 7fd8db7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd8dc133360 0x7fd8dc09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:55.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.349+0000 7fd8dbfff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd8dc132f50 0x7fd8dc09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50324/0 (socket says 172.21.15.103:50324) 2023-07-02T09:01:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.349+0000 7fd8db7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd8dc133360 0x7fd8dc09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60380/0 (socket says 172.21.15.103:60380) 2023-07-02T09:01:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.349+0000 7fd8dbfff700 1 -- 172.21.15.103:0/1478383418 learned_addr learned my addr 172.21.15.103:0/1478383418 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.350+0000 7fd8db7fe700 1 -- 172.21.15.103:0/1478383418 >> v1:172.21.15.103:6789/0 conn(0x7fd8dc132f50 legacy=0x7fd8dc09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.350+0000 7fd8db7fe700 1 -- 172.21.15.103:0/1478383418 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc132a20 con 0x7fd8dc133360 2023-07-02T09:01:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.350+0000 7fd8db7fe700 1 --2- 172.21.15.103:0/1478383418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc133360 0x7fd8dc09cc90 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25243c55475998f7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.350+0000 7fd8daffd700 1 -- 172.21.15.103:0/1478383418 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8c4009cd0 con 0x7fd8dc133360 2023-07-02T09:01:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.350+0000 7fd8daffd700 1 -- 172.21.15.103:0/1478383418 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd8c4009e30 con 0x7fd8dc133360 2023-07-02T09:01:55.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.350+0000 7fd8daffd700 1 -- 172.21.15.103:0/1478383418 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8c400e360 con 0x7fd8dc133360 2023-07-02T09:01:55.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.350+0000 7fd8e297f700 1 -- 172.21.15.103:0/1478383418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc133360 msgr2=0x7fd8dc09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:55.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.350+0000 7fd8e297f700 1 --2- 172.21.15.103:0/1478383418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc133360 0x7fd8dc09cc90 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:55.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.351+0000 7fd8e297f700 1 -- 172.21.15.103:0/1478383418 shutdown_connections 2023-07-02T09:01:55.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.351+0000 7fd8e297f700 1 --2- 172.21.15.103:0/1478383418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc133360 0x7fd8dc09cc90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.351+0000 7fd8e297f700 1 -- 172.21.15.103:0/1478383418 >> 172.21.15.103:0/1478383418 conn(0x7fd8dc0a2080 msgr2=0x7fd8dc0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.351+0000 7fd8e297f700 1 -- 172.21.15.103:0/1478383418 shutdown_connections 2023-07-02T09:01:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.351+0000 7fd8e297f700 1 -- 172.21.15.103:0/1478383418 wait complete. 2023-07-02T09:01:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.351+0000 7fd8e297f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.352+0000 7fd8e297f700 1 Processor -- start 2023-07-02T09:01:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.352+0000 7fd8e297f700 1 -- start start 2023-07-02T09:01:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8e297f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc132f50 0x7fd8dc12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8e297f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc06d260 con 0x7fd8dc132f50 2023-07-02T09:01:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc132f50 0x7fd8dc12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc132f50 0x7fd8dc12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60388/0 (socket says 172.21.15.103:60388) 2023-07-02T09:01:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8dbfff700 1 -- 172.21.15.103:0/2903510747 learned_addr learned my addr 172.21.15.103:0/2903510747 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8dbfff700 1 -- 172.21.15.103:0/2903510747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc12bb90 con 0x7fd8dc132f50 2023-07-02T09:01:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8dbfff700 1 --2- 172.21.15.103:0/2903510747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc132f50 0x7fd8dc12bea0 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8d8ff9700 1 -- 172.21.15.103:0/2903510747 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8cc005a60 con 0x7fd8dc132f50 2023-07-02T09:01:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8d8ff9700 1 -- 172.21.15.103:0/2903510747 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd8cc005bc0 con 0x7fd8dc132f50 2023-07-02T09:01:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8d8ff9700 1 -- 172.21.15.103:0/2903510747 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8cc005e60 con 0x7fd8dc132f50 2023-07-02T09:01:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc06f730 con 0x7fd8dc132f50 2023-07-02T09:01:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.353+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc06f490 con 0x7fd8dc132f50 2023-07-02T09:01:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.354+0000 7fd8d8ff9700 1 -- 172.21.15.103:0/2903510747 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd8cc0070b0 con 0x7fd8dc132f50 2023-07-02T09:01:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.354+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc061ea0 con 0x7fd8dc132f50 2023-07-02T09:01:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.354+0000 7fd8d8ff9700 1 --2- 172.21.15.103:0/2903510747 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8bc037810 0x7fd8bc039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.354+0000 7fd8d8ff9700 1 -- 172.21.15.103:0/2903510747 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd8cc046ab0 con 0x7fd8dc132f50 2023-07-02T09:01:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.355+0000 7fd8db7fe700 1 --2- 172.21.15.103:0/2903510747 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8bc037810 0x7fd8bc039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:55.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.355+0000 7fd8db7fe700 1 --2- 172.21.15.103:0/2903510747 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8bc037810 0x7fd8bc039d20 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:55.359 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.359+0000 7fd8d8ff9700 1 -- 172.21.15.103:0/2903510747 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd8cc00a4a0 con 0x7fd8dc132f50 2023-07-02T09:01:55.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.548+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd8dc06f490 con 0x7fd8dc132f50 2023-07-02T09:01:55.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.548+0000 7fd8d8ff9700 1 -- 172.21.15.103:0/2903510747 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd8dc06f490 con 0x7fd8dc132f50 2023-07-02T09:01:55.549 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.550+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8bc037810 msgr2=0x7fd8bc039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.550+0000 7fd8e297f700 1 --2- 172.21.15.103:0/2903510747 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8bc037810 0x7fd8bc039d20 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.550+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc132f50 msgr2=0x7fd8dc12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.550+0000 7fd8e297f700 1 --2- 172.21.15.103:0/2903510747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc132f50 0x7fd8dc12bea0 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.551+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 shutdown_connections 2023-07-02T09:01:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.551+0000 7fd8e297f700 1 --2- 172.21.15.103:0/2903510747 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8bc037810 0x7fd8bc039d20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.551+0000 7fd8e297f700 1 --2- 172.21.15.103:0/2903510747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8dc132f50 0x7fd8dc12bea0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.551+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 >> 172.21.15.103:0/2903510747 conn(0x7fd8dc0a2080 msgr2=0x7fd8dc1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.551+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 shutdown_connections 2023-07-02T09:01:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:55.551+0000 7fd8e297f700 1 -- 172.21.15.103:0/2903510747 wait complete. 2023-07-02T09:01:55.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:56.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:56.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.704+0000 7fb456a87700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:56.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.709+0000 7fb456a87700 1 Processor -- start 2023-07-02T09:01:56.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.710+0000 7fb456a87700 1 -- start start 2023-07-02T09:01:56.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.710+0000 7fb456a87700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb450133360 0x7fb45009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:56.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.710+0000 7fb456a87700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45009e260 con 0x7fb450132f50 2023-07-02T09:01:56.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.710+0000 7fb456a87700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4500a1640 con 0x7fb450133360 2023-07-02T09:01:56.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.710+0000 7fb44ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb450133360 0x7fb45009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:56.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.710+0000 7fb44ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb450133360 0x7fb45009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60398/0 (socket says 172.21.15.103:60398) 2023-07-02T09:01:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.710+0000 7fb44ffff700 1 -- 172.21.15.103:0/168298812 learned_addr learned my addr 172.21.15.103:0/168298812 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.710+0000 7fb454823700 1 --1- 172.21.15.103:0/168298812 >> v1:172.21.15.103:6789/0 conn(0x7fb450132f50 0x7fb45009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50330/0 (socket says 172.21.15.103:50330) 2023-07-02T09:01:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.711+0000 7fb44ffff700 1 -- 172.21.15.103:0/168298812 >> v1:172.21.15.103:6789/0 conn(0x7fb450132f50 legacy=0x7fb45009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.711+0000 7fb44ffff700 1 -- 172.21.15.103:0/168298812 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb450132a20 con 0x7fb450133360 2023-07-02T09:01:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.711+0000 7fb44ffff700 1 --2- 172.21.15.103:0/168298812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450133360 0x7fb45009cc90 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e753f38b53e84699 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.711+0000 7fb44f7fe700 1 -- 172.21.15.103:0/168298812 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb43c009d90 con 0x7fb450133360 2023-07-02T09:01:56.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.711+0000 7fb44f7fe700 1 -- 172.21.15.103:0/168298812 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb43c00e230 con 0x7fb450133360 2023-07-02T09:01:56.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.711+0000 7fb44f7fe700 1 -- 172.21.15.103:0/168298812 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb43c00e4d0 con 0x7fb450133360 2023-07-02T09:01:56.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.712+0000 7fb456a87700 1 -- 172.21.15.103:0/168298812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450133360 msgr2=0x7fb45009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:56.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.712+0000 7fb456a87700 1 --2- 172.21.15.103:0/168298812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450133360 0x7fb45009cc90 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:56.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.712+0000 7fb456a87700 1 -- 172.21.15.103:0/168298812 shutdown_connections 2023-07-02T09:01:56.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.712+0000 7fb456a87700 1 --2- 172.21.15.103:0/168298812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450133360 0x7fb45009cc90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:56.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.712+0000 7fb456a87700 1 -- 172.21.15.103:0/168298812 >> 172.21.15.103:0/168298812 conn(0x7fb4500a2080 msgr2=0x7fb4500a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.712+0000 7fb456a87700 1 -- 172.21.15.103:0/168298812 shutdown_connections 2023-07-02T09:01:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.712+0000 7fb456a87700 1 -- 172.21.15.103:0/168298812 wait complete. 2023-07-02T09:01:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.712+0000 7fb456a87700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.713+0000 7fb456a87700 1 Processor -- start 2023-07-02T09:01:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.713+0000 7fb456a87700 1 -- start start 2023-07-02T09:01:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb456a87700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450132f50 0x7fb45012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb456a87700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45006d260 con 0x7fb450132f50 2023-07-02T09:01:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb454823700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450132f50 0x7fb45012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb454823700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450132f50 0x7fb45012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60406/0 (socket says 172.21.15.103:60406) 2023-07-02T09:01:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb454823700 1 -- 172.21.15.103:0/3195083758 learned_addr learned my addr 172.21.15.103:0/3195083758 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb454823700 1 -- 172.21.15.103:0/3195083758 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45012bb90 con 0x7fb450132f50 2023-07-02T09:01:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb454823700 1 --2- 172.21.15.103:0/3195083758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450132f50 0x7fb45012bea0 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb44d7fa700 1 -- 172.21.15.103:0/3195083758 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb4440059a0 con 0x7fb450132f50 2023-07-02T09:01:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb44d7fa700 1 -- 172.21.15.103:0/3195083758 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb444005b00 con 0x7fb450132f50 2023-07-02T09:01:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb44d7fa700 1 -- 172.21.15.103:0/3195083758 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb444005da0 con 0x7fb450132f50 2023-07-02T09:01:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45006f730 con 0x7fb450132f50 2023-07-02T09:01:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.714+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45006f490 con 0x7fb450132f50 2023-07-02T09:01:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.715+0000 7fb44d7fa700 1 -- 172.21.15.103:0/3195083758 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb4440070b0 con 0x7fb450132f50 2023-07-02T09:01:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.715+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb450061ea0 con 0x7fb450132f50 2023-07-02T09:01:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.715+0000 7fb44d7fa700 1 --2- 172.21.15.103:0/3195083758 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb4340377c0 0x7fb434039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.715+0000 7fb44d7fa700 1 -- 172.21.15.103:0/3195083758 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb444046a70 con 0x7fb450132f50 2023-07-02T09:01:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.716+0000 7fb44ffff700 1 --2- 172.21.15.103:0/3195083758 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb4340377c0 0x7fb434039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:56.719 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.716+0000 7fb44ffff700 1 --2- 172.21.15.103:0/3195083758 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb4340377c0 0x7fb434039cd0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.720+0000 7fb44d7fa700 1 -- 172.21.15.103:0/3195083758 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb44400a6b0 con 0x7fb450132f50 2023-07-02T09:01:56.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.909+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb45006f490 con 0x7fb450132f50 2023-07-02T09:01:56.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.910+0000 7fb44d7fa700 1 -- 172.21.15.103:0/3195083758 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb45006f490 con 0x7fb450132f50 2023-07-02T09:01:56.910 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb4340377c0 msgr2=0x7fb434039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 --2- 172.21.15.103:0/3195083758 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb4340377c0 0x7fb434039cd0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450132f50 msgr2=0x7fb45012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 --2- 172.21.15.103:0/3195083758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450132f50 0x7fb45012bea0 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 shutdown_connections 2023-07-02T09:01:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 --2- 172.21.15.103:0/3195083758 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb4340377c0 0x7fb434039cd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 --2- 172.21.15.103:0/3195083758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb450132f50 0x7fb45012bea0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 >> 172.21.15.103:0/3195083758 conn(0x7fb4500a2080 msgr2=0x7fb4501bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 shutdown_connections 2023-07-02T09:01:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:56.912+0000 7fb456a87700 1 -- 172.21.15.103:0/3195083758 wait complete. 2023-07-02T09:01:56.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:57.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:57.975 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~132s 2023-07-02T09:01:58.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.065+0000 7f7622727700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:58.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.070+0000 7f7622727700 1 Processor -- start 2023-07-02T09:01:58.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.070+0000 7f7622727700 1 -- start start 2023-07-02T09:01:58.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f7622727700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f761c133360 0x7f761c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:58.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f7622727700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c09e260 con 0x7f761c132f50 2023-07-02T09:01:58.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f7622727700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761c0a1640 con 0x7f761c133360 2023-07-02T09:01:58.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f761c133360 0x7f761c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:58.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f761c133360 0x7f761c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60418/0 (socket says 172.21.15.103:60418) 2023-07-02T09:01:58.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761bfff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f761c132f50 0x7f761c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50340/0 (socket says 172.21.15.103:50340) 2023-07-02T09:01:58.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761b7fe700 1 -- 172.21.15.103:0/3791455926 learned_addr learned my addr 172.21.15.103:0/3791455926 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761b7fe700 1 -- 172.21.15.103:0/3791455926 >> v1:172.21.15.103:6789/0 conn(0x7f761c132f50 legacy=0x7f761c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761b7fe700 1 -- 172.21.15.103:0/3791455926 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c132a20 con 0x7f761c133360 2023-07-02T09:01:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761affd700 1 -- 172.21.15.103:0/3791455926 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160695929 0 0) 0x7f761c09e260 con 0x7f761c132f50 2023-07-02T09:01:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761b7fe700 1 --2- 172.21.15.103:0/3791455926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c133360 0x7f761c09cc90 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c281a4dc94b7f2 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.071+0000 7f761affd700 1 -- 172.21.15.103:0/3791455926 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f760c009d90 con 0x7f761c133360 2023-07-02T09:01:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f761affd700 1 -- 172.21.15.103:0/3791455926 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f760c00e230 con 0x7f761c133360 2023-07-02T09:01:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f761affd700 1 -- 172.21.15.103:0/3791455926 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f760c00e4d0 con 0x7f761c133360 2023-07-02T09:01:58.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f7622727700 1 -- 172.21.15.103:0/3791455926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c133360 msgr2=0x7f761c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:58.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f7622727700 1 --2- 172.21.15.103:0/3791455926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c133360 0x7f761c09cc90 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:58.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f7622727700 1 -- 172.21.15.103:0/3791455926 shutdown_connections 2023-07-02T09:01:58.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f7622727700 1 --2- 172.21.15.103:0/3791455926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c133360 0x7f761c09cc90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:58.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f7622727700 1 -- 172.21.15.103:0/3791455926 >> 172.21.15.103:0/3791455926 conn(0x7f761c0a2080 msgr2=0x7f761c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:58.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f7622727700 1 -- 172.21.15.103:0/3791455926 shutdown_connections 2023-07-02T09:01:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.072+0000 7f7622727700 1 -- 172.21.15.103:0/3791455926 wait complete. 2023-07-02T09:01:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.073+0000 7f7622727700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.073+0000 7f7622727700 1 Processor -- start 2023-07-02T09:01:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.074+0000 7f7622727700 1 -- start start 2023-07-02T09:01:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.074+0000 7f7622727700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c132f50 0x7f761c1ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.074+0000 7f7622727700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c06d370 con 0x7f761c132f50 2023-07-02T09:01:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.074+0000 7f761bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c132f50 0x7f761c1ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.074+0000 7f761bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c132f50 0x7f761c1ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60422/0 (socket says 172.21.15.103:60422) 2023-07-02T09:01:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.074+0000 7f761bfff700 1 -- 172.21.15.103:0/1044179974 learned_addr learned my addr 172.21.15.103:0/1044179974 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.075+0000 7f761bfff700 1 -- 172.21.15.103:0/1044179974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c06dde0 con 0x7f761c132f50 2023-07-02T09:01:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.075+0000 7f761bfff700 1 --2- 172.21.15.103:0/1044179974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c132f50 0x7f761c1ba9b0 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.075+0000 7f7618ff9700 1 -- 172.21.15.103:0/1044179974 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7604004b80 con 0x7f761c132f50 2023-07-02T09:01:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.075+0000 7f7618ff9700 1 -- 172.21.15.103:0/1044179974 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7604004ce0 con 0x7f761c132f50 2023-07-02T09:01:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.075+0000 7f7618ff9700 1 -- 172.21.15.103:0/1044179974 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f760400c4f0 con 0x7f761c132f50 2023-07-02T09:01:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.075+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c1ba4f0 con 0x7f761c132f50 2023-07-02T09:01:58.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.075+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c071e40 con 0x7f761c132f50 2023-07-02T09:01:58.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.076+0000 7f7618ff9700 1 -- 172.21.15.103:0/1044179974 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f760400c650 con 0x7f761c132f50 2023-07-02T09:01:58.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.076+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c061ea0 con 0x7f761c132f50 2023-07-02T09:01:58.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.076+0000 7f7618ff9700 1 --2- 172.21.15.103:0/1044179974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75fc0377c0 0x7f75fc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:58.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.076+0000 7f7618ff9700 1 -- 172.21.15.103:0/1044179974 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7604046970 con 0x7f761c132f50 2023-07-02T09:01:58.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.077+0000 7f761b7fe700 1 --2- 172.21.15.103:0/1044179974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75fc0377c0 0x7f75fc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:58.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.077+0000 7f761b7fe700 1 --2- 172.21.15.103:0/1044179974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75fc0377c0 0x7f75fc039cd0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:58.081 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.081+0000 7f7618ff9700 1 -- 172.21.15.103:0/1044179974 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7604009c80 con 0x7f761c132f50 2023-07-02T09:01:58.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.270+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f761c0725b0 con 0x7f761c132f50 2023-07-02T09:01:58.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.270+0000 7f7618ff9700 1 -- 172.21.15.103:0/1044179974 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f76040098a0 con 0x7f761c132f50 2023-07-02T09:01:58.271 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:58.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75fc0377c0 msgr2=0x7f75fc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:58.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 --2- 172.21.15.103:0/1044179974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75fc0377c0 0x7f75fc039cd0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:58.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c132f50 msgr2=0x7f761c1ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:58.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 --2- 172.21.15.103:0/1044179974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c132f50 0x7f761c1ba9b0 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:58.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 shutdown_connections 2023-07-02T09:01:58.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 --2- 172.21.15.103:0/1044179974 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f75fc0377c0 0x7f75fc039cd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:58.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 --2- 172.21.15.103:0/1044179974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f761c132f50 0x7f761c1ba9b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 >> 172.21.15.103:0/1044179974 conn(0x7f761c0a2080 msgr2=0x7f761c0a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 shutdown_connections 2023-07-02T09:01:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:58.272+0000 7f7622727700 1 -- 172.21.15.103:0/1044179974 wait complete. 2023-07-02T09:01:58.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:01:59.287 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:01:59.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.418+0000 7fdc27ed3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:59.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.423+0000 7fdc27ed3700 1 Processor -- start 2023-07-02T09:01:59.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.423+0000 7fdc27ed3700 1 -- start start 2023-07-02T09:01:59.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc27ed3700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdc2012f250 0x7fdc2012f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:59.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc27ed3700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2006df60 con 0x7fdc20098a00 2023-07-02T09:01:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc27ed3700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2009a710 con 0x7fdc2012f250 2023-07-02T09:01:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc25c6f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdc2012f250 0x7fdc2012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc2546e700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdc20098a00 0x7fdc20098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50346/0 (socket says 172.21.15.103:50346) 2023-07-02T09:01:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc2546e700 1 -- 172.21.15.103:0/851607453 learned_addr learned my addr 172.21.15.103:0/851607453 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:01:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc25c6f700 1 -- 172.21.15.103:0/851607453 >> v1:172.21.15.103:6789/0 conn(0x7fdc20098a00 legacy=0x7fdc20098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc25c6f700 1 -- 172.21.15.103:0/851607453 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2012ed20 con 0x7fdc2012f250 2023-07-02T09:01:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc25c6f700 1 --2- 172.21.15.103:0/851607453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc2012f250 0x7fdc2012f660 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bcc845df0db1dcf server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc24c6d700 1 -- 172.21.15.103:0/851607453 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc1c00d450 con 0x7fdc2012f250 2023-07-02T09:01:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc24c6d700 1 -- 172.21.15.103:0/851607453 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdc1c00d5b0 con 0x7fdc2012f250 2023-07-02T09:01:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.424+0000 7fdc24c6d700 1 -- 172.21.15.103:0/851607453 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc1c00d850 con 0x7fdc2012f250 2023-07-02T09:01:59.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.425+0000 7fdc27ed3700 1 -- 172.21.15.103:0/851607453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc2012f250 msgr2=0x7fdc2012f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:59.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.425+0000 7fdc27ed3700 1 --2- 172.21.15.103:0/851607453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc2012f250 0x7fdc2012f660 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:59.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.425+0000 7fdc27ed3700 1 -- 172.21.15.103:0/851607453 shutdown_connections 2023-07-02T09:01:59.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.425+0000 7fdc27ed3700 1 --2- 172.21.15.103:0/851607453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc2012f250 0x7fdc2012f660 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:59.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.425+0000 7fdc27ed3700 1 -- 172.21.15.103:0/851607453 >> 172.21.15.103:0/851607453 conn(0x7fdc2009d810 msgr2=0x7fdc2009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:59.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.425+0000 7fdc27ed3700 1 -- 172.21.15.103:0/851607453 shutdown_connections 2023-07-02T09:01:59.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.425+0000 7fdc27ed3700 1 -- 172.21.15.103:0/851607453 wait complete. 2023-07-02T09:01:59.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.425+0000 7fdc27ed3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:01:59.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.426+0000 7fdc27ed3700 1 Processor -- start 2023-07-02T09:01:59.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.426+0000 7fdc27ed3700 1 -- start start 2023-07-02T09:01:59.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.427+0000 7fdc27ed3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc20098a00 0x7fdc20117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:59.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.427+0000 7fdc27ed3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2011a9f0 con 0x7fdc20098a00 2023-07-02T09:01:59.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.427+0000 7fdc25c6f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc20098a00 0x7fdc20117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:59.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.427+0000 7fdc25c6f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc20098a00 0x7fdc20117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60428/0 (socket says 172.21.15.103:60428) 2023-07-02T09:01:59.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.427+0000 7fdc25c6f700 1 -- 172.21.15.103:0/707127942 learned_addr learned my addr 172.21.15.103:0/707127942 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:01:59.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.427+0000 7fdc25c6f700 1 -- 172.21.15.103:0/707127942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20118f50 con 0x7fdc20098a00 2023-07-02T09:01:59.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.427+0000 7fdc25c6f700 1 --2- 172.21.15.103:0/707127942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc20098a00 0x7fdc20117360 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:59.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.428+0000 7fdc167fc700 1 -- 172.21.15.103:0/707127942 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc1c00c990 con 0x7fdc20098a00 2023-07-02T09:01:59.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.428+0000 7fdc167fc700 1 -- 172.21.15.103:0/707127942 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdc1c00caf0 con 0x7fdc20098a00 2023-07-02T09:01:59.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.428+0000 7fdc167fc700 1 -- 172.21.15.103:0/707127942 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdc1c00cd90 con 0x7fdc20098a00 2023-07-02T09:01:59.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.428+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2011a650 con 0x7fdc20098a00 2023-07-02T09:01:59.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.428+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2011afe0 con 0x7fdc20098a00 2023-07-02T09:01:59.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.428+0000 7fdc167fc700 1 -- 172.21.15.103:0/707127942 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fdc1c015040 con 0x7fdc20098a00 2023-07-02T09:01:59.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.429+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2011a840 con 0x7fdc20098a00 2023-07-02T09:01:59.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.429+0000 7fdc167fc700 1 --2- 172.21.15.103:0/707127942 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc040377c0 0x7fdc04039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:01:59.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.429+0000 7fdc167fc700 1 -- 172.21.15.103:0/707127942 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fdc1c012040 con 0x7fdc20098a00 2023-07-02T09:01:59.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.429+0000 7fdc2546e700 1 --2- 172.21.15.103:0/707127942 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc040377c0 0x7fdc04039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:01:59.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.430+0000 7fdc2546e700 1 --2- 172.21.15.103:0/707127942 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc040377c0 0x7fdc04039cd0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:01:59.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.434+0000 7fdc167fc700 1 -- 172.21.15.103:0/707127942 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fdc1c014c50 con 0x7fdc20098a00 2023-07-02T09:01:59.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.624+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdc201198a0 con 0x7fdc20098a00 2023-07-02T09:01:59.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.625+0000 7fdc167fc700 1 -- 172.21.15.103:0/707127942 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fdc1c014a70 con 0x7fdc20098a00 2023-07-02T09:01:59.625 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:01:59.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc040377c0 msgr2=0x7fdc04039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:59.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 --2- 172.21.15.103:0/707127942 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc040377c0 0x7fdc04039cd0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:59.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc20098a00 msgr2=0x7fdc20117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:01:59.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 --2- 172.21.15.103:0/707127942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc20098a00 0x7fdc20117360 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:59.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 shutdown_connections 2023-07-02T09:01:59.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 --2- 172.21.15.103:0/707127942 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdc040377c0 0x7fdc04039cd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:59.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 --2- 172.21.15.103:0/707127942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc20098a00 0x7fdc20117360 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:01:59.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 >> 172.21.15.103:0/707127942 conn(0x7fdc2009d810 msgr2=0x7fdc2009f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:01:59.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 shutdown_connections 2023-07-02T09:01:59.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:01:59.627+0000 7fdc27ed3700 1 -- 172.21.15.103:0/707127942 wait complete. 2023-07-02T09:01:59.640 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:00.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:00.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.779+0000 7f485553d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:00.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.784+0000 7f485553d700 1 Processor -- start 2023-07-02T09:02:00.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.784+0000 7f485553d700 1 -- start start 2023-07-02T09:02:00.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f485553d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4850132f50 0x7f4850133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:00.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f485553d700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485009e260 con 0x7f485009c580 2023-07-02T09:02:00.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f485553d700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48500a1640 con 0x7f4850132f50 2023-07-02T09:02:00.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f484effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4850132f50 0x7f4850133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:00.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f484effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4850132f50 0x7f4850133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60442/0 (socket says 172.21.15.103:60442) 2023-07-02T09:02:00.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f484e7fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f485009c580 0x7f485009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50356/0 (socket says 172.21.15.103:50356) 2023-07-02T09:02:00.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f484effd700 1 -- 172.21.15.103:0/1912043999 learned_addr learned my addr 172.21.15.103:0/1912043999 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:00.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f484effd700 1 -- 172.21.15.103:0/1912043999 >> v1:172.21.15.103:6789/0 conn(0x7f485009c580 legacy=0x7f485009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:00.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.785+0000 7f484effd700 1 -- 172.21.15.103:0/1912043999 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4850132a20 con 0x7f4850132f50 2023-07-02T09:02:00.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f484effd700 1 --2- 172.21.15.103:0/1912043999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4850132f50 0x7f4850133360 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3beb05c896115cd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:00.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f484dffb700 1 -- 172.21.15.103:0/1912043999 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4844009d90 con 0x7f4850132f50 2023-07-02T09:02:00.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f484dffb700 1 -- 172.21.15.103:0/1912043999 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f484400e230 con 0x7f4850132f50 2023-07-02T09:02:00.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f484dffb700 1 -- 172.21.15.103:0/1912043999 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f484400e4d0 con 0x7f4850132f50 2023-07-02T09:02:00.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f485553d700 1 -- 172.21.15.103:0/1912043999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4850132f50 msgr2=0x7f4850133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:00.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f485553d700 1 --2- 172.21.15.103:0/1912043999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4850132f50 0x7f4850133360 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:00.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f485553d700 1 -- 172.21.15.103:0/1912043999 shutdown_connections 2023-07-02T09:02:00.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f485553d700 1 --2- 172.21.15.103:0/1912043999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4850132f50 0x7f4850133360 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:00.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f485553d700 1 -- 172.21.15.103:0/1912043999 >> 172.21.15.103:0/1912043999 conn(0x7f48500a2080 msgr2=0x7f48500a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:00.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f485553d700 1 -- 172.21.15.103:0/1912043999 shutdown_connections 2023-07-02T09:02:00.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.786+0000 7f485553d700 1 -- 172.21.15.103:0/1912043999 wait complete. 2023-07-02T09:02:00.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.787+0000 7f485553d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:00.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.788+0000 7f485553d700 1 Processor -- start 2023-07-02T09:02:00.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.788+0000 7f485553d700 1 -- start start 2023-07-02T09:02:00.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.788+0000 7f485553d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485009c580 0x7f485012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:00.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.788+0000 7f485553d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485006f9d0 con 0x7f485009c580 2023-07-02T09:02:00.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f484effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485009c580 0x7f485012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:00.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f484effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485009c580 0x7f485012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60452/0 (socket says 172.21.15.103:60452) 2023-07-02T09:02:00.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f484effd700 1 -- 172.21.15.103:0/3540467877 learned_addr learned my addr 172.21.15.103:0/3540467877 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:00.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f484effd700 1 -- 172.21.15.103:0/3540467877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4850133a00 con 0x7f485009c580 2023-07-02T09:02:00.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f484effd700 1 --2- 172.21.15.103:0/3540467877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485009c580 0x7f485012b7b0 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:00.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f48377fe700 1 -- 172.21.15.103:0/3540467877 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4844014540 con 0x7f485009c580 2023-07-02T09:02:00.792 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f48377fe700 1 -- 172.21.15.103:0/3540467877 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f48440146a0 con 0x7f485009c580 2023-07-02T09:02:00.792 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485006d160 con 0x7f485009c580 2023-07-02T09:02:00.792 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f48377fe700 1 -- 172.21.15.103:0/3540467877 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4844014940 con 0x7f485009c580 2023-07-02T09:02:00.792 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.789+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485006da70 con 0x7f485009c580 2023-07-02T09:02:00.793 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.790+0000 7f48377fe700 1 -- 172.21.15.103:0/3540467877 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4844014af0 con 0x7f485009c580 2023-07-02T09:02:00.793 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.790+0000 7f48377fe700 1 --2- 172.21.15.103:0/3540467877 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f483c0377b0 0x7f483c039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:00.793 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.790+0000 7f48377fe700 1 -- 172.21.15.103:0/3540467877 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4844048350 con 0x7f485009c580 2023-07-02T09:02:00.793 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.790+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4850061ea0 con 0x7f485009c580 2023-07-02T09:02:00.793 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.790+0000 7f484e7fc700 1 --2- 172.21.15.103:0/3540467877 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f483c0377b0 0x7f483c039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:00.793 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.791+0000 7f484e7fc700 1 --2- 172.21.15.103:0/3540467877 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f483c0377b0 0x7f483c039cc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:00.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.795+0000 7f48377fe700 1 -- 172.21.15.103:0/3540467877 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f484400d1c0 con 0x7f485009c580 2023-07-02T09:02:00.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.984+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f48500725b0 con 0x7f485009c580 2023-07-02T09:02:00.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.984+0000 7f48377fe700 1 -- 172.21.15.103:0/3540467877 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4844011320 con 0x7f485009c580 2023-07-02T09:02:00.985 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:00.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f483c0377b0 msgr2=0x7f483c039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:00.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 --2- 172.21.15.103:0/3540467877 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f483c0377b0 0x7f483c039cc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:00.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485009c580 msgr2=0x7f485012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:00.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 --2- 172.21.15.103:0/3540467877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485009c580 0x7f485012b7b0 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:00.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 shutdown_connections 2023-07-02T09:02:00.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 --2- 172.21.15.103:0/3540467877 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f483c0377b0 0x7f483c039cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:00.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 --2- 172.21.15.103:0/3540467877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f485009c580 0x7f485012b7b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:00.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 >> 172.21.15.103:0/3540467877 conn(0x7f48500a2080 msgr2=0x7f48501babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:00.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 shutdown_connections 2023-07-02T09:02:00.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:00.987+0000 7f485553d700 1 -- 172.21.15.103:0/3540467877 wait complete. 2023-07-02T09:02:01.000 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:02.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:02.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.140+0000 7fadb9359700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:02.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.146+0000 7fadb9359700 1 Processor -- start 2023-07-02T09:02:02.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.146+0000 7fadb9359700 1 -- start start 2023-07-02T09:02:02.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb9359700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fadb412fb10 0x7fadb412ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:02.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb9359700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb412f750 con 0x7fadb406bf60 2023-07-02T09:02:02.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb9359700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb409abe0 con 0x7fadb412fb10 2023-07-02T09:02:02.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb2ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fadb412fb10 0x7fadb412ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:02.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb27fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fadb406bf60 0x7fadb406c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50368/0 (socket says 172.21.15.103:50368) 2023-07-02T09:02:02.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb2ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fadb412fb10 0x7fadb412ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60456/0 (socket says 172.21.15.103:60456) 2023-07-02T09:02:02.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb27fc700 1 -- 172.21.15.103:0/759149252 learned_addr learned my addr 172.21.15.103:0/759149252 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:02.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb2ffd700 1 -- 172.21.15.103:0/759149252 >> v1:172.21.15.103:6789/0 conn(0x7fadb406bf60 legacy=0x7fadb406c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:02.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb2ffd700 1 -- 172.21.15.103:0/759149252 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb4066840 con 0x7fadb412fb10 2023-07-02T09:02:02.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb1ffb700 1 -- 172.21.15.103:0/759149252 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760145412 0 0) 0x7fadb412f750 con 0x7fadb406bf60 2023-07-02T09:02:02.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb2ffd700 1 --2- 172.21.15.103:0/759149252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb412fb10 0x7fadb412ff20 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11ec79a130cd719 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:02.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb1ffb700 1 -- 172.21.15.103:0/759149252 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fada8005d80 con 0x7fadb412fb10 2023-07-02T09:02:02.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb1ffb700 1 -- 172.21.15.103:0/759149252 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fada8009930 con 0x7fadb412fb10 2023-07-02T09:02:02.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.147+0000 7fadb1ffb700 1 -- 172.21.15.103:0/759149252 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fada8009bd0 con 0x7fadb412fb10 2023-07-02T09:02:02.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.148+0000 7fadb9359700 1 -- 172.21.15.103:0/759149252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb412fb10 msgr2=0x7fadb412ff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:02.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.148+0000 7fadb9359700 1 --2- 172.21.15.103:0/759149252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb412fb10 0x7fadb412ff20 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:02.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.148+0000 7fadb9359700 1 -- 172.21.15.103:0/759149252 shutdown_connections 2023-07-02T09:02:02.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.148+0000 7fadb9359700 1 --2- 172.21.15.103:0/759149252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb412fb10 0x7fadb412ff20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:02.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.148+0000 7fadb9359700 1 -- 172.21.15.103:0/759149252 >> 172.21.15.103:0/759149252 conn(0x7fadb40a0100 msgr2=0x7fadb40a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:02.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.148+0000 7fadb9359700 1 -- 172.21.15.103:0/759149252 shutdown_connections 2023-07-02T09:02:02.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.148+0000 7fadb9359700 1 -- 172.21.15.103:0/759149252 wait complete. 2023-07-02T09:02:02.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.148+0000 7fadb9359700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:02.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.149+0000 7fadb9359700 1 Processor -- start 2023-07-02T09:02:02.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.149+0000 7fadb9359700 1 -- start start 2023-07-02T09:02:02.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.150+0000 7fadb9359700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb406bf60 0x7fadb41c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:02.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.150+0000 7fadb9359700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4120cb0 con 0x7fadb406bf60 2023-07-02T09:02:02.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.150+0000 7fadb2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb406bf60 0x7fadb41c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:02.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.150+0000 7fadb2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb406bf60 0x7fadb41c71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60460/0 (socket says 172.21.15.103:60460) 2023-07-02T09:02:02.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.150+0000 7fadb2ffd700 1 -- 172.21.15.103:0/802196520 learned_addr learned my addr 172.21.15.103:0/802196520 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:02.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.150+0000 7fadb2ffd700 1 -- 172.21.15.103:0/802196520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb411fce0 con 0x7fadb406bf60 2023-07-02T09:02:02.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.151+0000 7fadb2ffd700 1 --2- 172.21.15.103:0/802196520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb406bf60 0x7fadb41c71a0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.151+0000 7fad9b7fe700 1 -- 172.21.15.103:0/802196520 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fada8014540 con 0x7fadb406bf60 2023-07-02T09:02:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.151+0000 7fad9b7fe700 1 -- 172.21.15.103:0/802196520 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fada80146a0 con 0x7fadb406bf60 2023-07-02T09:02:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.151+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb4120a10 con 0x7fadb406bf60 2023-07-02T09:02:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.151+0000 7fad9b7fe700 1 -- 172.21.15.103:0/802196520 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fada8014940 con 0x7fadb406bf60 2023-07-02T09:02:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.151+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb411f390 con 0x7fadb406bf60 2023-07-02T09:02:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.152+0000 7fad9b7fe700 1 -- 172.21.15.103:0/802196520 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fada8014af0 con 0x7fadb406bf60 2023-07-02T09:02:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.152+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb404e800 con 0x7fadb406bf60 2023-07-02T09:02:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.152+0000 7fad9b7fe700 1 --2- 172.21.15.103:0/802196520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fada00377c0 0x7fada0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:02.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.152+0000 7fad9b7fe700 1 -- 172.21.15.103:0/802196520 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fada8047e40 con 0x7fadb406bf60 2023-07-02T09:02:02.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.152+0000 7fadb27fc700 1 --2- 172.21.15.103:0/802196520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fada00377c0 0x7fada0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:02.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.153+0000 7fadb27fc700 1 --2- 172.21.15.103:0/802196520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fada00377c0 0x7fada0039cd0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:02.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.156+0000 7fad9b7fe700 1 -- 172.21.15.103:0/802196520 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fada8006850 con 0x7fadb406bf60 2023-07-02T09:02:02.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.347+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fadb4061ea0 con 0x7fadb406bf60 2023-07-02T09:02:02.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.347+0000 7fad9b7fe700 1 -- 172.21.15.103:0/802196520 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fada8006440 con 0x7fadb406bf60 2023-07-02T09:02:02.348 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:02.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fada00377c0 msgr2=0x7fada0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:02.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 --2- 172.21.15.103:0/802196520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fada00377c0 0x7fada0039cd0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:02.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb406bf60 msgr2=0x7fadb41c71a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:02.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 --2- 172.21.15.103:0/802196520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb406bf60 0x7fadb41c71a0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:02.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 shutdown_connections 2023-07-02T09:02:02.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 --2- 172.21.15.103:0/802196520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fada00377c0 0x7fada0039cd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:02.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 --2- 172.21.15.103:0/802196520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb406bf60 0x7fadb41c71a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:02.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 >> 172.21.15.103:0/802196520 conn(0x7fadb40a0100 msgr2=0x7fadb4132aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:02.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 shutdown_connections 2023-07-02T09:02:02.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:02.350+0000 7fadb9359700 1 -- 172.21.15.103:0/802196520 wait complete. 2023-07-02T09:02:02.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:03.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:03.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~138s 2023-07-02T09:02:03.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.569+0000 7f489b6b8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:03.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.574+0000 7f489b6b8700 1 Processor -- start 2023-07-02T09:02:03.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.574+0000 7f489b6b8700 1 -- start start 2023-07-02T09:02:03.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f489b6b8700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4894133360 0x7f489409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:03.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f489b6b8700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489409e260 con 0x7f4894132f50 2023-07-02T09:02:03.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f489b6b8700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48940a1640 con 0x7f4894133360 2023-07-02T09:02:03.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f4898c53700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4894133360 0x7f489409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:03.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f4898c53700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4894133360 0x7f489409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41904/0 (socket says 172.21.15.103:41904) 2023-07-02T09:02:03.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f4898c53700 1 -- 172.21.15.103:0/881825563 learned_addr learned my addr 172.21.15.103:0/881825563 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:03.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f4899454700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4894132f50 0x7f489409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54748/0 (socket says 172.21.15.103:54748) 2023-07-02T09:02:03.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f4898c53700 1 -- 172.21.15.103:0/881825563 >> v1:172.21.15.103:6789/0 conn(0x7f4894132f50 legacy=0x7f489409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:03.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f4898c53700 1 -- 172.21.15.103:0/881825563 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4894132a20 con 0x7f4894133360 2023-07-02T09:02:03.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f4898c53700 1 --2- 172.21.15.103:0/881825563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894133360 0x7f489409cc90 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ec391e3f863c48 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:03.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.575+0000 7f488bfff700 1 -- 172.21.15.103:0/881825563 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4884009d90 con 0x7f4894133360 2023-07-02T09:02:03.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f488bfff700 1 -- 172.21.15.103:0/881825563 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f488400e230 con 0x7f4894133360 2023-07-02T09:02:03.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f488bfff700 1 -- 172.21.15.103:0/881825563 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f488400e4d0 con 0x7f4894133360 2023-07-02T09:02:03.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f489b6b8700 1 -- 172.21.15.103:0/881825563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894133360 msgr2=0x7f489409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:03.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f489b6b8700 1 --2- 172.21.15.103:0/881825563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894133360 0x7f489409cc90 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:03.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f489b6b8700 1 -- 172.21.15.103:0/881825563 shutdown_connections 2023-07-02T09:02:03.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f489b6b8700 1 --2- 172.21.15.103:0/881825563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894133360 0x7f489409cc90 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:03.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f489b6b8700 1 -- 172.21.15.103:0/881825563 >> 172.21.15.103:0/881825563 conn(0x7f48940a2080 msgr2=0x7f48940a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:03.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f489b6b8700 1 -- 172.21.15.103:0/881825563 shutdown_connections 2023-07-02T09:02:03.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.576+0000 7f489b6b8700 1 -- 172.21.15.103:0/881825563 wait complete. 2023-07-02T09:02:03.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.577+0000 7f489b6b8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:03.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.577+0000 7f489b6b8700 1 Processor -- start 2023-07-02T09:02:03.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.578+0000 7f489b6b8700 1 -- start start 2023-07-02T09:02:03.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.578+0000 7f489b6b8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894132f50 0x7f489412bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:03.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.578+0000 7f489b6b8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489406d260 con 0x7f4894132f50 2023-07-02T09:02:03.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.578+0000 7f4899454700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894132f50 0x7f489412bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:03.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.578+0000 7f4899454700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894132f50 0x7f489412bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41906/0 (socket says 172.21.15.103:41906) 2023-07-02T09:02:03.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.578+0000 7f4899454700 1 -- 172.21.15.103:0/1873475392 learned_addr learned my addr 172.21.15.103:0/1873475392 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:03.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.579+0000 7f4899454700 1 -- 172.21.15.103:0/1873475392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489412bb90 con 0x7f4894132f50 2023-07-02T09:02:03.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.579+0000 7f4899454700 1 --2- 172.21.15.103:0/1873475392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894132f50 0x7f489412bea0 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:03.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.579+0000 7f4889ffb700 1 -- 172.21.15.103:0/1873475392 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f48900059a0 con 0x7f4894132f50 2023-07-02T09:02:03.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.579+0000 7f4889ffb700 1 -- 172.21.15.103:0/1873475392 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4890005b00 con 0x7f4894132f50 2023-07-02T09:02:03.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.579+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489406f730 con 0x7f4894132f50 2023-07-02T09:02:03.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.579+0000 7f4889ffb700 1 -- 172.21.15.103:0/1873475392 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4890005da0 con 0x7f4894132f50 2023-07-02T09:02:03.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.579+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489406f490 con 0x7f4894132f50 2023-07-02T09:02:03.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.580+0000 7f4889ffb700 1 -- 172.21.15.103:0/1873475392 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f48900070b0 con 0x7f4894132f50 2023-07-02T09:02:03.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.580+0000 7f4889ffb700 1 --2- 172.21.15.103:0/1873475392 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48780377c0 0x7f4878039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:03.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.580+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4894061ea0 con 0x7f4894132f50 2023-07-02T09:02:03.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.580+0000 7f4889ffb700 1 -- 172.21.15.103:0/1873475392 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f48900469f0 con 0x7f4894132f50 2023-07-02T09:02:03.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.580+0000 7f4898c53700 1 --2- 172.21.15.103:0/1873475392 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48780377c0 0x7f4878039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:03.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.581+0000 7f4898c53700 1 --2- 172.21.15.103:0/1873475392 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48780377c0 0x7f4878039cd0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:03.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.585+0000 7f4889ffb700 1 -- 172.21.15.103:0/1873475392 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f489000a6b0 con 0x7f4894132f50 2023-07-02T09:02:03.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.778+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f489406f490 con 0x7f4894132f50 2023-07-02T09:02:03.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.779+0000 7f4889ffb700 1 -- 172.21.15.103:0/1873475392 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f489406f490 con 0x7f4894132f50 2023-07-02T09:02:03.779 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:03.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48780377c0 msgr2=0x7f4878039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:03.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 --2- 172.21.15.103:0/1873475392 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48780377c0 0x7f4878039cd0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:03.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894132f50 msgr2=0x7f489412bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:03.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 --2- 172.21.15.103:0/1873475392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894132f50 0x7f489412bea0 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:03.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 shutdown_connections 2023-07-02T09:02:03.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 --2- 172.21.15.103:0/1873475392 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f48780377c0 0x7f4878039cd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:03.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 --2- 172.21.15.103:0/1873475392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4894132f50 0x7f489412bea0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:03.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 >> 172.21.15.103:0/1873475392 conn(0x7f48940a2080 msgr2=0x7f48941bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:03.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 shutdown_connections 2023-07-02T09:02:03.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:03.781+0000 7f489b6b8700 1 -- 172.21.15.103:0/1873475392 wait complete. 2023-07-02T09:02:03.794 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:04.795 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:04.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.934+0000 7fd8d6175700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:04.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.939+0000 7fd8d6175700 1 Processor -- start 2023-07-02T09:02:04.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.939+0000 7fd8d6175700 1 -- start start 2023-07-02T09:02:04.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.940+0000 7fd8d6175700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd8d0133360 0x7fd8d009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:04.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.940+0000 7fd8d6175700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d009e260 con 0x7fd8d0132f50 2023-07-02T09:02:04.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.940+0000 7fd8d6175700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d00a1640 con 0x7fd8d0133360 2023-07-02T09:02:04.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.940+0000 7fd8ceffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd8d0133360 0x7fd8d009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:04.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.940+0000 7fd8cf7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd8d0132f50 0x7fd8d009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54754/0 (socket says 172.21.15.103:54754) 2023-07-02T09:02:04.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.940+0000 7fd8cf7fe700 1 -- 172.21.15.103:0/1685489050 learned_addr learned my addr 172.21.15.103:0/1685489050 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:04.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.940+0000 7fd8ceffd700 1 -- 172.21.15.103:0/1685489050 >> v1:172.21.15.103:6789/0 conn(0x7fd8d0132f50 legacy=0x7fd8d009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:04.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.940+0000 7fd8ceffd700 1 -- 172.21.15.103:0/1685489050 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d0132a20 con 0x7fd8d0133360 2023-07-02T09:02:04.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8ceffd700 1 --2- 172.21.15.103:0/1685489050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0133360 0x7fd8d009cc90 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5c4134920a0ca0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:04.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8ce7fc700 1 -- 172.21.15.103:0/1685489050 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8c0009cd0 con 0x7fd8d0133360 2023-07-02T09:02:04.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8ce7fc700 1 -- 172.21.15.103:0/1685489050 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd8c0009e30 con 0x7fd8d0133360 2023-07-02T09:02:04.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8ce7fc700 1 -- 172.21.15.103:0/1685489050 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8c000e360 con 0x7fd8d0133360 2023-07-02T09:02:04.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8d6175700 1 -- 172.21.15.103:0/1685489050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0133360 msgr2=0x7fd8d009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:04.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8d6175700 1 --2- 172.21.15.103:0/1685489050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0133360 0x7fd8d009cc90 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:04.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8d6175700 1 -- 172.21.15.103:0/1685489050 shutdown_connections 2023-07-02T09:02:04.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8d6175700 1 --2- 172.21.15.103:0/1685489050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0133360 0x7fd8d009cc90 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:04.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8d6175700 1 -- 172.21.15.103:0/1685489050 >> 172.21.15.103:0/1685489050 conn(0x7fd8d00a2080 msgr2=0x7fd8d00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:04.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.941+0000 7fd8d6175700 1 -- 172.21.15.103:0/1685489050 shutdown_connections 2023-07-02T09:02:04.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.942+0000 7fd8d6175700 1 -- 172.21.15.103:0/1685489050 wait complete. 2023-07-02T09:02:04.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.942+0000 7fd8d6175700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:04.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.943+0000 7fd8d6175700 1 Processor -- start 2023-07-02T09:02:04.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.943+0000 7fd8d6175700 1 -- start start 2023-07-02T09:02:04.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8d6175700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0132f50 0x7fd8d012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:04.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8d6175700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d006d260 con 0x7fd8d0132f50 2023-07-02T09:02:04.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8cf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0132f50 0x7fd8d012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:04.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8cf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0132f50 0x7fd8d012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41930/0 (socket says 172.21.15.103:41930) 2023-07-02T09:02:04.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8cf7fe700 1 -- 172.21.15.103:0/2513070889 learned_addr learned my addr 172.21.15.103:0/2513070889 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:04.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8cf7fe700 1 -- 172.21.15.103:0/2513070889 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d012bb90 con 0x7fd8d0132f50 2023-07-02T09:02:04.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8cf7fe700 1 --2- 172.21.15.103:0/2513070889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0132f50 0x7fd8d012bea0 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:04.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8c7fff700 1 -- 172.21.15.103:0/2513070889 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8b8005a60 con 0x7fd8d0132f50 2023-07-02T09:02:04.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8c7fff700 1 -- 172.21.15.103:0/2513070889 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd8b8005bc0 con 0x7fd8d0132f50 2023-07-02T09:02:04.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d006f730 con 0x7fd8d0132f50 2023-07-02T09:02:04.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8c7fff700 1 -- 172.21.15.103:0/2513070889 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd8b8005e60 con 0x7fd8d0132f50 2023-07-02T09:02:04.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.944+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d006f490 con 0x7fd8d0132f50 2023-07-02T09:02:04.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.945+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d0061ea0 con 0x7fd8d0132f50 2023-07-02T09:02:04.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.945+0000 7fd8c7fff700 1 -- 172.21.15.103:0/2513070889 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd8b80070b0 con 0x7fd8d0132f50 2023-07-02T09:02:04.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.945+0000 7fd8c7fff700 1 --2- 172.21.15.103:0/2513070889 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8b00377c0 0x7fd8b0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:04.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.945+0000 7fd8c7fff700 1 -- 172.21.15.103:0/2513070889 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd8b80469f0 con 0x7fd8d0132f50 2023-07-02T09:02:04.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.946+0000 7fd8ceffd700 1 --2- 172.21.15.103:0/2513070889 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8b00377c0 0x7fd8b0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:04.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.946+0000 7fd8ceffd700 1 --2- 172.21.15.103:0/2513070889 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8b00377c0 0x7fd8b0039cd0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:04.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:04.950+0000 7fd8c7fff700 1 -- 172.21.15.103:0/2513070889 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd8b800a4a0 con 0x7fd8d0132f50 2023-07-02T09:02:05.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.137+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd8d006f490 con 0x7fd8d0132f50 2023-07-02T09:02:05.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.137+0000 7fd8c7fff700 1 -- 172.21.15.103:0/2513070889 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd8d006f490 con 0x7fd8d0132f50 2023-07-02T09:02:05.138 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:05.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8b00377c0 msgr2=0x7fd8b0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:05.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 --2- 172.21.15.103:0/2513070889 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8b00377c0 0x7fd8b0039cd0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:05.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0132f50 msgr2=0x7fd8d012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:05.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 --2- 172.21.15.103:0/2513070889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0132f50 0x7fd8d012bea0 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:05.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 shutdown_connections 2023-07-02T09:02:05.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 --2- 172.21.15.103:0/2513070889 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd8b00377c0 0x7fd8b0039cd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:05.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 --2- 172.21.15.103:0/2513070889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0132f50 0x7fd8d012bea0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:05.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 >> 172.21.15.103:0/2513070889 conn(0x7fd8d00a2080 msgr2=0x7fd8d01bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:05.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 shutdown_connections 2023-07-02T09:02:05.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:05.139+0000 7fd8d6175700 1 -- 172.21.15.103:0/2513070889 wait complete. 2023-07-02T09:02:05.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:06.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:06.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.294+0000 7f1605aea700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:06.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.299+0000 7f1605aea700 1 Processor -- start 2023-07-02T09:02:06.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.299+0000 7f1605aea700 1 -- start start 2023-07-02T09:02:06.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f1605aea700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1600133360 0x7f160009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:06.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f1605aea700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160009e260 con 0x7f1600132f50 2023-07-02T09:02:06.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f1605aea700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16000a1640 con 0x7f1600133360 2023-07-02T09:02:06.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15feffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1600133360 0x7f160009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:06.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15ff7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1600132f50 0x7f160009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54764/0 (socket says 172.21.15.103:54764) 2023-07-02T09:02:06.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15feffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1600133360 0x7f160009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41934/0 (socket says 172.21.15.103:41934) 2023-07-02T09:02:06.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15ff7fe700 1 -- 172.21.15.103:0/1901154302 learned_addr learned my addr 172.21.15.103:0/1901154302 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:06.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15feffd700 1 -- 172.21.15.103:0/1901154302 >> v1:172.21.15.103:6789/0 conn(0x7f1600132f50 legacy=0x7f160009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15feffd700 1 -- 172.21.15.103:0/1901154302 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1600132a20 con 0x7f1600133360 2023-07-02T09:02:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15feffd700 1 --2- 172.21.15.103:0/1901154302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600133360 0x7f160009cc90 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed655e397e8e131 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15fe7fc700 1 -- 172.21.15.103:0/1901154302 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f15f0009cd0 con 0x7f1600133360 2023-07-02T09:02:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.300+0000 7f15fe7fc700 1 -- 172.21.15.103:0/1901154302 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f15f0009e30 con 0x7f1600133360 2023-07-02T09:02:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f15fe7fc700 1 -- 172.21.15.103:0/1901154302 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f15f000e360 con 0x7f1600133360 2023-07-02T09:02:06.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f1605aea700 1 -- 172.21.15.103:0/1901154302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600133360 msgr2=0x7f160009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:06.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f1605aea700 1 --2- 172.21.15.103:0/1901154302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600133360 0x7f160009cc90 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:06.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f1605aea700 1 -- 172.21.15.103:0/1901154302 shutdown_connections 2023-07-02T09:02:06.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f1605aea700 1 --2- 172.21.15.103:0/1901154302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600133360 0x7f160009cc90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:06.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f1605aea700 1 -- 172.21.15.103:0/1901154302 >> 172.21.15.103:0/1901154302 conn(0x7f16000a2080 msgr2=0x7f16000a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:06.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f1605aea700 1 -- 172.21.15.103:0/1901154302 shutdown_connections 2023-07-02T09:02:06.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f1605aea700 1 -- 172.21.15.103:0/1901154302 wait complete. 2023-07-02T09:02:06.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.301+0000 7f1605aea700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:06.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.302+0000 7f1605aea700 1 Processor -- start 2023-07-02T09:02:06.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.303+0000 7f1605aea700 1 -- start start 2023-07-02T09:02:06.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.303+0000 7f1605aea700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600132f50 0x7f160012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:06.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.303+0000 7f1605aea700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160006d260 con 0x7f1600132f50 2023-07-02T09:02:06.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.303+0000 7f15ff7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600132f50 0x7f160012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:06.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.303+0000 7f15ff7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600132f50 0x7f160012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41948/0 (socket says 172.21.15.103:41948) 2023-07-02T09:02:06.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.303+0000 7f15ff7fe700 1 -- 172.21.15.103:0/1322734507 learned_addr learned my addr 172.21.15.103:0/1322734507 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:06.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.304+0000 7f15ff7fe700 1 -- 172.21.15.103:0/1322734507 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160012bb90 con 0x7f1600132f50 2023-07-02T09:02:06.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.304+0000 7f15ff7fe700 1 --2- 172.21.15.103:0/1322734507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600132f50 0x7f160012bea0 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:06.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.304+0000 7f15f7fff700 1 -- 172.21.15.103:0/1322734507 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f15e8005a60 con 0x7f1600132f50 2023-07-02T09:02:06.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.304+0000 7f15f7fff700 1 -- 172.21.15.103:0/1322734507 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f15e8005bc0 con 0x7f1600132f50 2023-07-02T09:02:06.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.304+0000 7f15f7fff700 1 -- 172.21.15.103:0/1322734507 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f15e8005e60 con 0x7f1600132f50 2023-07-02T09:02:06.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.304+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160006f730 con 0x7f1600132f50 2023-07-02T09:02:06.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.304+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160006f490 con 0x7f1600132f50 2023-07-02T09:02:06.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.305+0000 7f15f7fff700 1 -- 172.21.15.103:0/1322734507 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f15e80070b0 con 0x7f1600132f50 2023-07-02T09:02:06.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.305+0000 7f15f7fff700 1 --2- 172.21.15.103:0/1322734507 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f15e00377c0 0x7f15e0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:06.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.305+0000 7f15f7fff700 1 -- 172.21.15.103:0/1322734507 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f15e80469f0 con 0x7f1600132f50 2023-07-02T09:02:06.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.305+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160006f490 con 0x7f1600132f50 2023-07-02T09:02:06.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.305+0000 7f15feffd700 1 --2- 172.21.15.103:0/1322734507 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f15e00377c0 0x7f15e0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:06.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.306+0000 7f15feffd700 1 --2- 172.21.15.103:0/1322734507 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f15e00377c0 0x7f15e0039cd0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:06.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.310+0000 7f15f7fff700 1 -- 172.21.15.103:0/1322734507 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f160006f490 con 0x7f1600132f50 2023-07-02T09:02:06.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.502+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f160006f490 con 0x7f1600132f50 2023-07-02T09:02:06.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.502+0000 7f15f7fff700 1 -- 172.21.15.103:0/1322734507 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f160006f490 con 0x7f1600132f50 2023-07-02T09:02:06.502 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:06.504 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f15e00377c0 msgr2=0x7f15e0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:06.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 --2- 172.21.15.103:0/1322734507 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f15e00377c0 0x7f15e0039cd0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:06.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600132f50 msgr2=0x7f160012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:06.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 --2- 172.21.15.103:0/1322734507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600132f50 0x7f160012bea0 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:06.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 shutdown_connections 2023-07-02T09:02:06.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 --2- 172.21.15.103:0/1322734507 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f15e00377c0 0x7f15e0039cd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:06.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 --2- 172.21.15.103:0/1322734507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1600132f50 0x7f160012bea0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:06.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 >> 172.21.15.103:0/1322734507 conn(0x7f16000a2080 msgr2=0x7f16001bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:06.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 shutdown_connections 2023-07-02T09:02:06.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:06.504+0000 7f1605aea700 1 -- 172.21.15.103:0/1322734507 wait complete. 2023-07-02T09:02:06.517 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:07.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:07.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.657+0000 7fd239ae9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:07.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd239ae9700 1 Processor -- start 2023-07-02T09:02:07.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd239ae9700 1 -- start start 2023-07-02T09:02:07.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd239ae9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd234133360 0x7fd23409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:07.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd239ae9700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23409e260 con 0x7fd234132f50 2023-07-02T09:02:07.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd239ae9700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2340a1640 con 0x7fd234133360 2023-07-02T09:02:07.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd232ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd234133360 0x7fd23409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:07.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd232ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd234133360 0x7fd23409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41958/0 (socket says 172.21.15.103:41958) 2023-07-02T09:02:07.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd2337fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd234132f50 0x7fd23409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54778/0 (socket says 172.21.15.103:54778) 2023-07-02T09:02:07.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.663+0000 7fd232ffd700 1 -- 172.21.15.103:0/261254156 learned_addr learned my addr 172.21.15.103:0/261254156 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:07.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.664+0000 7fd232ffd700 1 -- 172.21.15.103:0/261254156 >> v1:172.21.15.103:6789/0 conn(0x7fd234132f50 legacy=0x7fd23409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:07.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.664+0000 7fd232ffd700 1 -- 172.21.15.103:0/261254156 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd234132a20 con 0x7fd234133360 2023-07-02T09:02:07.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.664+0000 7fd2327fc700 1 -- 172.21.15.103:0/261254156 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98369350 0 0) 0x7fd23409e260 con 0x7fd234132f50 2023-07-02T09:02:07.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.664+0000 7fd232ffd700 1 --2- 172.21.15.103:0/261254156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234133360 0x7fd23409cc90 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ee40f13a1abec4 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:07.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.664+0000 7fd2327fc700 1 -- 172.21.15.103:0/261254156 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd224009d90 con 0x7fd234133360 2023-07-02T09:02:07.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.664+0000 7fd2327fc700 1 -- 172.21.15.103:0/261254156 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd22400e230 con 0x7fd234133360 2023-07-02T09:02:07.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.664+0000 7fd2327fc700 1 -- 172.21.15.103:0/261254156 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd22400e4d0 con 0x7fd234133360 2023-07-02T09:02:07.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.665+0000 7fd239ae9700 1 -- 172.21.15.103:0/261254156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234133360 msgr2=0x7fd23409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:07.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.665+0000 7fd239ae9700 1 --2- 172.21.15.103:0/261254156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234133360 0x7fd23409cc90 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:07.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.665+0000 7fd239ae9700 1 -- 172.21.15.103:0/261254156 shutdown_connections 2023-07-02T09:02:07.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.665+0000 7fd239ae9700 1 --2- 172.21.15.103:0/261254156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234133360 0x7fd23409cc90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:07.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.665+0000 7fd239ae9700 1 -- 172.21.15.103:0/261254156 >> 172.21.15.103:0/261254156 conn(0x7fd2340a2080 msgr2=0x7fd2340a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:07.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.665+0000 7fd239ae9700 1 -- 172.21.15.103:0/261254156 shutdown_connections 2023-07-02T09:02:07.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.665+0000 7fd239ae9700 1 -- 172.21.15.103:0/261254156 wait complete. 2023-07-02T09:02:07.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.665+0000 7fd239ae9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:07.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.666+0000 7fd239ae9700 1 Processor -- start 2023-07-02T09:02:07.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.666+0000 7fd239ae9700 1 -- start start 2023-07-02T09:02:07.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd239ae9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234132f50 0x7fd2341ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:07.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd239ae9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23406d370 con 0x7fd234132f50 2023-07-02T09:02:07.670 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd2337fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234132f50 0x7fd2341ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:07.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd2337fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234132f50 0x7fd2341ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41972/0 (socket says 172.21.15.103:41972) 2023-07-02T09:02:07.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd2337fe700 1 -- 172.21.15.103:0/2723516959 learned_addr learned my addr 172.21.15.103:0/2723516959 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:07.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd2337fe700 1 -- 172.21.15.103:0/2723516959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23406dde0 con 0x7fd234132f50 2023-07-02T09:02:07.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd2337fe700 1 --2- 172.21.15.103:0/2723516959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234132f50 0x7fd2341ba9b0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:07.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd22bfff700 1 -- 172.21.15.103:0/2723516959 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd21c004b80 con 0x7fd234132f50 2023-07-02T09:02:07.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd22bfff700 1 -- 172.21.15.103:0/2723516959 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd21c004ce0 con 0x7fd234132f50 2023-07-02T09:02:07.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2341ba4f0 con 0x7fd234132f50 2023-07-02T09:02:07.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.667+0000 7fd22bfff700 1 -- 172.21.15.103:0/2723516959 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd21c00c4f0 con 0x7fd234132f50 2023-07-02T09:02:07.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.668+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd234071e40 con 0x7fd234132f50 2023-07-02T09:02:07.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.668+0000 7fd22bfff700 1 -- 172.21.15.103:0/2723516959 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd21c00c650 con 0x7fd234132f50 2023-07-02T09:02:07.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.668+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd234061ea0 con 0x7fd234132f50 2023-07-02T09:02:07.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.669+0000 7fd22bfff700 1 --2- 172.21.15.103:0/2723516959 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd214037810 0x7fd214039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:07.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.669+0000 7fd22bfff700 1 -- 172.21.15.103:0/2723516959 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd21c046a30 con 0x7fd234132f50 2023-07-02T09:02:07.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.669+0000 7fd232ffd700 1 --2- 172.21.15.103:0/2723516959 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd214037810 0x7fd214039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:07.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.669+0000 7fd232ffd700 1 --2- 172.21.15.103:0/2723516959 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd214037810 0x7fd214039d20 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:07.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.673+0000 7fd22bfff700 1 -- 172.21.15.103:0/2723516959 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd21c009c80 con 0x7fd234132f50 2023-07-02T09:02:07.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.866+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd2340725b0 con 0x7fd234132f50 2023-07-02T09:02:07.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.866+0000 7fd22bfff700 1 -- 172.21.15.103:0/2723516959 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd21c0098a0 con 0x7fd234132f50 2023-07-02T09:02:07.867 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:07.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd214037810 msgr2=0x7fd214039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:07.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 --2- 172.21.15.103:0/2723516959 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd214037810 0x7fd214039d20 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:07.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234132f50 msgr2=0x7fd2341ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:07.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 --2- 172.21.15.103:0/2723516959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234132f50 0x7fd2341ba9b0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:07.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 shutdown_connections 2023-07-02T09:02:07.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 --2- 172.21.15.103:0/2723516959 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd214037810 0x7fd214039d20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:07.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 --2- 172.21.15.103:0/2723516959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd234132f50 0x7fd2341ba9b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:07.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 >> 172.21.15.103:0/2723516959 conn(0x7fd2340a2080 msgr2=0x7fd2340a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:07.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 shutdown_connections 2023-07-02T09:02:07.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:07.869+0000 7fd239ae9700 1 -- 172.21.15.103:0/2723516959 wait complete. 2023-07-02T09:02:07.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:08.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:08.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~143s 2023-07-02T09:02:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.022+0000 7fb35ec8b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.027+0000 7fb35ec8b700 1 Processor -- start 2023-07-02T09:02:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.027+0000 7fb35ec8b700 1 -- start start 2023-07-02T09:02:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb35ec8b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb358133360 0x7fb35809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb35ec8b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35809e260 con 0x7fb358132f50 2023-07-02T09:02:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb35ec8b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3580a1640 con 0x7fb358133360 2023-07-02T09:02:09.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb357fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb358133360 0x7fb35809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:09.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb35ca27700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb358132f50 0x7fb35809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54794/0 (socket says 172.21.15.103:54794) 2023-07-02T09:02:09.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb35ca27700 1 -- 172.21.15.103:0/3969998613 learned_addr learned my addr 172.21.15.103:0/3969998613 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:09.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb357fff700 1 -- 172.21.15.103:0/3969998613 >> v1:172.21.15.103:6789/0 conn(0x7fb358132f50 legacy=0x7fb35809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:09.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb357fff700 1 -- 172.21.15.103:0/3969998613 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358132a20 con 0x7fb358133360 2023-07-02T09:02:09.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.028+0000 7fb357fff700 1 --2- 172.21.15.103:0/3969998613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358133360 0x7fb35809cc90 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d051ec162380e688 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:09.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb3577fe700 1 -- 172.21.15.103:0/3969998613 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb344009cd0 con 0x7fb358133360 2023-07-02T09:02:09.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb3577fe700 1 -- 172.21.15.103:0/3969998613 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb344009e30 con 0x7fb358133360 2023-07-02T09:02:09.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb3577fe700 1 -- 172.21.15.103:0/3969998613 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb34400e360 con 0x7fb358133360 2023-07-02T09:02:09.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb35ec8b700 1 -- 172.21.15.103:0/3969998613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358133360 msgr2=0x7fb35809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:09.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb35ec8b700 1 --2- 172.21.15.103:0/3969998613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358133360 0x7fb35809cc90 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:09.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb35ec8b700 1 -- 172.21.15.103:0/3969998613 shutdown_connections 2023-07-02T09:02:09.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb35ec8b700 1 --2- 172.21.15.103:0/3969998613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358133360 0x7fb35809cc90 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:09.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb35ec8b700 1 -- 172.21.15.103:0/3969998613 >> 172.21.15.103:0/3969998613 conn(0x7fb3580a2080 msgr2=0x7fb3580a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:09.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb35ec8b700 1 -- 172.21.15.103:0/3969998613 shutdown_connections 2023-07-02T09:02:09.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.029+0000 7fb35ec8b700 1 -- 172.21.15.103:0/3969998613 wait complete. 2023-07-02T09:02:09.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.030+0000 7fb35ec8b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:09.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.030+0000 7fb35ec8b700 1 Processor -- start 2023-07-02T09:02:09.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.031+0000 7fb35ec8b700 1 -- start start 2023-07-02T09:02:09.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.031+0000 7fb35ec8b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358132f50 0x7fb35812bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:09.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.031+0000 7fb35ec8b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35806d260 con 0x7fb358132f50 2023-07-02T09:02:09.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.031+0000 7fb35ca27700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358132f50 0x7fb35812bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:09.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.031+0000 7fb35ca27700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358132f50 0x7fb35812bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41992/0 (socket says 172.21.15.103:41992) 2023-07-02T09:02:09.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.031+0000 7fb35ca27700 1 -- 172.21.15.103:0/96330645 learned_addr learned my addr 172.21.15.103:0/96330645 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:09.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.032+0000 7fb35ca27700 1 -- 172.21.15.103:0/96330645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35812bb90 con 0x7fb358132f50 2023-07-02T09:02:09.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.032+0000 7fb35ca27700 1 --2- 172.21.15.103:0/96330645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358132f50 0x7fb35812bea0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:09.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.032+0000 7fb3557fa700 1 -- 172.21.15.103:0/96330645 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb34c005a60 con 0x7fb358132f50 2023-07-02T09:02:09.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.032+0000 7fb3557fa700 1 -- 172.21.15.103:0/96330645 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb34c005bc0 con 0x7fb358132f50 2023-07-02T09:02:09.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.032+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35806f730 con 0x7fb358132f50 2023-07-02T09:02:09.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.032+0000 7fb3557fa700 1 -- 172.21.15.103:0/96330645 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb34c005e60 con 0x7fb358132f50 2023-07-02T09:02:09.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.032+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35806f490 con 0x7fb358132f50 2023-07-02T09:02:09.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.033+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb358061ea0 con 0x7fb358132f50 2023-07-02T09:02:09.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.033+0000 7fb3557fa700 1 -- 172.21.15.103:0/96330645 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb34c0070b0 con 0x7fb358132f50 2023-07-02T09:02:09.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.033+0000 7fb3557fa700 1 --2- 172.21.15.103:0/96330645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb33c0377c0 0x7fb33c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.033+0000 7fb3557fa700 1 -- 172.21.15.103:0/96330645 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb34c0469f0 con 0x7fb358132f50 2023-07-02T09:02:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.034+0000 7fb357fff700 1 --2- 172.21.15.103:0/96330645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb33c0377c0 0x7fb33c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:09.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.034+0000 7fb357fff700 1 --2- 172.21.15.103:0/96330645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb33c0377c0 0x7fb33c039cd0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:09.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.038+0000 7fb3557fa700 1 -- 172.21.15.103:0/96330645 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb34c00a4a0 con 0x7fb358132f50 2023-07-02T09:02:09.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.231+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb35806f490 con 0x7fb358132f50 2023-07-02T09:02:09.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.231+0000 7fb3557fa700 1 -- 172.21.15.103:0/96330645 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb35806f490 con 0x7fb358132f50 2023-07-02T09:02:09.231 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:09.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb33c0377c0 msgr2=0x7fb33c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:09.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 --2- 172.21.15.103:0/96330645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb33c0377c0 0x7fb33c039cd0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:09.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358132f50 msgr2=0x7fb35812bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:09.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 --2- 172.21.15.103:0/96330645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358132f50 0x7fb35812bea0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:09.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 shutdown_connections 2023-07-02T09:02:09.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 --2- 172.21.15.103:0/96330645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb33c0377c0 0x7fb33c039cd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:09.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 --2- 172.21.15.103:0/96330645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb358132f50 0x7fb35812bea0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:09.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 >> 172.21.15.103:0/96330645 conn(0x7fb3580a2080 msgr2=0x7fb3581bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:09.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 shutdown_connections 2023-07-02T09:02:09.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:09.233+0000 7fb35ec8b700 1 -- 172.21.15.103:0/96330645 wait complete. 2023-07-02T09:02:09.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:10.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:10.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.387+0000 7fc3b43b2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:10.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.392+0000 7fc3b43b2700 1 Processor -- start 2023-07-02T09:02:10.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.392+0000 7fc3b43b2700 1 -- start start 2023-07-02T09:02:10.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.392+0000 7fc3b43b2700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc3ac132030 0x7fc3ac134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:10.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b43b2700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac06df60 con 0x7fc3ac12f250 2023-07-02T09:02:10.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b43b2700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ac09a710 con 0x7fc3ac132030 2023-07-02T09:02:10.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b194d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc3ac132030 0x7fc3ac134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:10.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b194d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc3ac132030 0x7fc3ac134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41996/0 (socket says 172.21.15.103:41996) 2023-07-02T09:02:10.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b214e700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc3ac12f250 0x7fc3ac12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54810/0 (socket says 172.21.15.103:54810) 2023-07-02T09:02:10.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b194d700 1 -- 172.21.15.103:0/2470735702 learned_addr learned my addr 172.21.15.103:0/2470735702 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:10.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b194d700 1 -- 172.21.15.103:0/2470735702 >> v1:172.21.15.103:6789/0 conn(0x7fc3ac12f250 legacy=0x7fc3ac12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:10.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b194d700 1 -- 172.21.15.103:0/2470735702 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac12ed20 con 0x7fc3ac132030 2023-07-02T09:02:10.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b194d700 1 --2- 172.21.15.103:0/2470735702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac132030 0x7fc3ac134450 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e7f03bc2fc3bac server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:10.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b114c700 1 -- 172.21.15.103:0/2470735702 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc3a800ad20 con 0x7fc3ac132030 2023-07-02T09:02:10.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b114c700 1 -- 172.21.15.103:0/2470735702 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc3a800ae80 con 0x7fc3ac132030 2023-07-02T09:02:10.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.393+0000 7fc3b114c700 1 -- 172.21.15.103:0/2470735702 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc3a800f470 con 0x7fc3ac132030 2023-07-02T09:02:10.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.394+0000 7fc3b43b2700 1 -- 172.21.15.103:0/2470735702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac132030 msgr2=0x7fc3ac134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:10.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.394+0000 7fc3b43b2700 1 --2- 172.21.15.103:0/2470735702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac132030 0x7fc3ac134450 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:10.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.394+0000 7fc3b43b2700 1 -- 172.21.15.103:0/2470735702 shutdown_connections 2023-07-02T09:02:10.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.394+0000 7fc3b43b2700 1 --2- 172.21.15.103:0/2470735702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac132030 0x7fc3ac134450 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:10.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.394+0000 7fc3b43b2700 1 -- 172.21.15.103:0/2470735702 >> 172.21.15.103:0/2470735702 conn(0x7fc3ac09d810 msgr2=0x7fc3ac09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:10.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.394+0000 7fc3b43b2700 1 -- 172.21.15.103:0/2470735702 shutdown_connections 2023-07-02T09:02:10.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.394+0000 7fc3b43b2700 1 -- 172.21.15.103:0/2470735702 wait complete. 2023-07-02T09:02:10.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.394+0000 7fc3b43b2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:10.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.395+0000 7fc3b43b2700 1 Processor -- start 2023-07-02T09:02:10.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.395+0000 7fc3b43b2700 1 -- start start 2023-07-02T09:02:10.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.396+0000 7fc3b43b2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac12f250 0x7fc3ac070260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:10.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.396+0000 7fc3b43b2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac071620 con 0x7fc3ac12f250 2023-07-02T09:02:10.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.396+0000 7fc3b214e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac12f250 0x7fc3ac070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:10.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.396+0000 7fc3b214e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac12f250 0x7fc3ac070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42004/0 (socket says 172.21.15.103:42004) 2023-07-02T09:02:10.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.396+0000 7fc3b214e700 1 -- 172.21.15.103:0/1363329286 learned_addr learned my addr 172.21.15.103:0/1363329286 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:10.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.396+0000 7fc3b214e700 1 -- 172.21.15.103:0/1363329286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac071ec0 con 0x7fc3ac12f250 2023-07-02T09:02:10.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.396+0000 7fc3b214e700 1 --2- 172.21.15.103:0/1363329286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac12f250 0x7fc3ac070260 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:10.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.397+0000 7fc39effd700 1 -- 172.21.15.103:0/1363329286 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc3a00059a0 con 0x7fc3ac12f250 2023-07-02T09:02:10.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.397+0000 7fc39effd700 1 -- 172.21.15.103:0/1363329286 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc3a0005b00 con 0x7fc3ac12f250 2023-07-02T09:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.397+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ac071310 con 0x7fc3ac12f250 2023-07-02T09:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.397+0000 7fc39effd700 1 -- 172.21.15.103:0/1363329286 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc3a0005da0 con 0x7fc3ac12f250 2023-07-02T09:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.397+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac11b0b0 con 0x7fc3ac12f250 2023-07-02T09:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.398+0000 7fc39effd700 1 -- 172.21.15.103:0/1363329286 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fc3a00070b0 con 0x7fc3ac12f250 2023-07-02T09:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.397+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ac061ea0 con 0x7fc3ac12f250 2023-07-02T09:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.398+0000 7fc39effd700 1 --2- 172.21.15.103:0/1363329286 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3900377c0 0x7fc390039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.398+0000 7fc39effd700 1 -- 172.21.15.103:0/1363329286 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc3a0046bc0 con 0x7fc3ac12f250 2023-07-02T09:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.398+0000 7fc3b194d700 1 --2- 172.21.15.103:0/1363329286 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3900377c0 0x7fc390039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:10.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.399+0000 7fc3b194d700 1 --2- 172.21.15.103:0/1363329286 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3900377c0 0x7fc390039cd0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:10.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.402+0000 7fc39effd700 1 -- 172.21.15.103:0/1363329286 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc3a000fe20 con 0x7fc3ac12f250 2023-07-02T09:02:10.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.596+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc3ac12c6c0 con 0x7fc3ac12f250 2023-07-02T09:02:10.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.596+0000 7fc39effd700 1 -- 172.21.15.103:0/1363329286 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fc3a000fe20 con 0x7fc3ac12f250 2023-07-02T09:02:10.597 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:10.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3900377c0 msgr2=0x7fc390039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:10.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 --2- 172.21.15.103:0/1363329286 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3900377c0 0x7fc390039cd0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:10.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac12f250 msgr2=0x7fc3ac070260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:10.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 --2- 172.21.15.103:0/1363329286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac12f250 0x7fc3ac070260 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:10.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 shutdown_connections 2023-07-02T09:02:10.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 --2- 172.21.15.103:0/1363329286 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3900377c0 0x7fc390039cd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:10.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 --2- 172.21.15.103:0/1363329286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3ac12f250 0x7fc3ac070260 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:10.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 >> 172.21.15.103:0/1363329286 conn(0x7fc3ac09d810 msgr2=0x7fc3ac12e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:10.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 shutdown_connections 2023-07-02T09:02:10.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:10.598+0000 7fc3b43b2700 1 -- 172.21.15.103:0/1363329286 wait complete. 2023-07-02T09:02:10.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:11.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:11.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.752+0000 7fdf2c867700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:11.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.757+0000 7fdf2c867700 1 Processor -- start 2023-07-02T09:02:11.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.757+0000 7fdf2c867700 1 -- start start 2023-07-02T09:02:11.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2c867700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf2812f250 0x7fdf2812f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:11.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2c867700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2806df60 con 0x7fdf28098a00 2023-07-02T09:02:11.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2c867700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2809a710 con 0x7fdf2812f250 2023-07-02T09:02:11.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf2812f250 0x7fdf2812f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:11.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf2812f250 0x7fdf2812f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42014/0 (socket says 172.21.15.103:42014) 2023-07-02T09:02:11.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf25d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdf28098a00 0x7fdf28098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54826/0 (socket says 172.21.15.103:54826) 2023-07-02T09:02:11.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2659c700 1 -- 172.21.15.103:0/917504348 learned_addr learned my addr 172.21.15.103:0/917504348 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:11.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2659c700 1 -- 172.21.15.103:0/917504348 >> v1:172.21.15.103:6789/0 conn(0x7fdf28098a00 legacy=0x7fdf28098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:11.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2659c700 1 -- 172.21.15.103:0/917504348 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2812ed20 con 0x7fdf2812f250 2023-07-02T09:02:11.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2659c700 1 --2- 172.21.15.103:0/917504348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf2812f250 0x7fdf2812f660 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87abcc7da14fab7c server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:11.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2559a700 1 -- 172.21.15.103:0/917504348 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf1c00d580 con 0x7fdf2812f250 2023-07-02T09:02:11.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2559a700 1 -- 172.21.15.103:0/917504348 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdf1c00d6e0 con 0x7fdf2812f250 2023-07-02T09:02:11.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.758+0000 7fdf2559a700 1 -- 172.21.15.103:0/917504348 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf1c00d980 con 0x7fdf2812f250 2023-07-02T09:02:11.762 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.759+0000 7fdf2c867700 1 -- 172.21.15.103:0/917504348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf2812f250 msgr2=0x7fdf2812f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:11.762 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.759+0000 7fdf2c867700 1 --2- 172.21.15.103:0/917504348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf2812f250 0x7fdf2812f660 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:11.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.759+0000 7fdf2c867700 1 -- 172.21.15.103:0/917504348 shutdown_connections 2023-07-02T09:02:11.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.759+0000 7fdf2c867700 1 --2- 172.21.15.103:0/917504348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf2812f250 0x7fdf2812f660 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:11.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.759+0000 7fdf2c867700 1 -- 172.21.15.103:0/917504348 >> 172.21.15.103:0/917504348 conn(0x7fdf2809d810 msgr2=0x7fdf2809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:11.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.759+0000 7fdf2c867700 1 -- 172.21.15.103:0/917504348 shutdown_connections 2023-07-02T09:02:11.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.759+0000 7fdf2c867700 1 -- 172.21.15.103:0/917504348 wait complete. 2023-07-02T09:02:11.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.759+0000 7fdf2c867700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:11.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.760+0000 7fdf2c867700 1 Processor -- start 2023-07-02T09:02:11.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.760+0000 7fdf2c867700 1 -- start start 2023-07-02T09:02:11.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.761+0000 7fdf2c867700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf28098a00 0x7fdf28117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:11.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.761+0000 7fdf2c867700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2811ad80 con 0x7fdf28098a00 2023-07-02T09:02:11.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.761+0000 7fdf2659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf28098a00 0x7fdf28117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:11.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.761+0000 7fdf2659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf28098a00 0x7fdf28117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42028/0 (socket says 172.21.15.103:42028) 2023-07-02T09:02:11.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.761+0000 7fdf2659c700 1 -- 172.21.15.103:0/4144279804 learned_addr learned my addr 172.21.15.103:0/4144279804 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:11.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.761+0000 7fdf2659c700 1 -- 172.21.15.103:0/4144279804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2811af70 con 0x7fdf28098a00 2023-07-02T09:02:11.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.762+0000 7fdf2659c700 1 --2- 172.21.15.103:0/4144279804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf28098a00 0x7fdf28117360 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:11.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.762+0000 7fdf16ffd700 1 -- 172.21.15.103:0/4144279804 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf1c00c950 con 0x7fdf28098a00 2023-07-02T09:02:11.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.762+0000 7fdf16ffd700 1 -- 172.21.15.103:0/4144279804 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdf1c00cab0 con 0x7fdf28098a00 2023-07-02T09:02:11.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.762+0000 7fdf16ffd700 1 -- 172.21.15.103:0/4144279804 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf1c00cd50 con 0x7fdf28098a00 2023-07-02T09:02:11.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.762+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf28119e60 con 0x7fdf28098a00 2023-07-02T09:02:11.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.762+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf28119b70 con 0x7fdf28098a00 2023-07-02T09:02:11.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.763+0000 7fdf16ffd700 1 -- 172.21.15.103:0/4144279804 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fdf1c015040 con 0x7fdf28098a00 2023-07-02T09:02:11.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.763+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2811b4c0 con 0x7fdf28098a00 2023-07-02T09:02:11.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.763+0000 7fdf16ffd700 1 --2- 172.21.15.103:0/4144279804 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf080377c0 0x7fdf08039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:11.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.763+0000 7fdf16ffd700 1 -- 172.21.15.103:0/4144279804 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fdf1c012040 con 0x7fdf28098a00 2023-07-02T09:02:11.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.763+0000 7fdf25d9b700 1 --2- 172.21.15.103:0/4144279804 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf080377c0 0x7fdf08039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:11.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.764+0000 7fdf25d9b700 1 --2- 172.21.15.103:0/4144279804 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf080377c0 0x7fdf08039cd0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:11.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.768+0000 7fdf16ffd700 1 -- 172.21.15.103:0/4144279804 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fdf1c013e10 con 0x7fdf28098a00 2023-07-02T09:02:11.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.962+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdf28061ea0 con 0x7fdf28098a00 2023-07-02T09:02:11.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.963+0000 7fdf16ffd700 1 -- 172.21.15.103:0/4144279804 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fdf1c013a00 con 0x7fdf28098a00 2023-07-02T09:02:11.963 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:11.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf080377c0 msgr2=0x7fdf08039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:11.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 --2- 172.21.15.103:0/4144279804 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf080377c0 0x7fdf08039cd0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:11.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf28098a00 msgr2=0x7fdf28117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:11.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 --2- 172.21.15.103:0/4144279804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf28098a00 0x7fdf28117360 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:11.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 shutdown_connections 2023-07-02T09:02:11.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 --2- 172.21.15.103:0/4144279804 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf080377c0 0x7fdf08039cd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:11.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 --2- 172.21.15.103:0/4144279804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf28098a00 0x7fdf28117360 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:11.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 >> 172.21.15.103:0/4144279804 conn(0x7fdf2809d810 msgr2=0x7fdf2812b520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:11.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 shutdown_connections 2023-07-02T09:02:11.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:11.965+0000 7fdf2c867700 1 -- 172.21.15.103:0/4144279804 wait complete. 2023-07-02T09:02:11.979 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:12.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:13.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.119+0000 7f03c83fa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:13.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.124+0000 7f03c83fa700 1 Processor -- start 2023-07-02T09:02:13.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.124+0000 7f03c83fa700 1 -- start start 2023-07-02T09:02:13.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c83fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f03c0132f50 0x7f03c0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:13.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c83fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c009e260 con 0x7f03c009c580 2023-07-02T09:02:13.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c83fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c00a1640 con 0x7f03c0132f50 2023-07-02T09:02:13.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c6196700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f03c0132f50 0x7f03c0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:13.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c5995700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f03c009c580 0x7f03c009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54840/0 (socket says 172.21.15.103:54840) 2023-07-02T09:02:13.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c6196700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f03c0132f50 0x7f03c0133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42044/0 (socket says 172.21.15.103:42044) 2023-07-02T09:02:13.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c5995700 1 -- 172.21.15.103:0/711785203 learned_addr learned my addr 172.21.15.103:0/711785203 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:13.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c6196700 1 -- 172.21.15.103:0/711785203 >> v1:172.21.15.103:6789/0 conn(0x7f03c009c580 legacy=0x7f03c009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:13.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.125+0000 7f03c6196700 1 -- 172.21.15.103:0/711785203 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c0132a20 con 0x7f03c0132f50 2023-07-02T09:02:13.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c6196700 1 --2- 172.21.15.103:0/711785203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c0132f50 0x7f03c0133360 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3027b2973bdd9957 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:13.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c5194700 1 -- 172.21.15.103:0/711785203 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f03b0009cd0 con 0x7f03c0132f50 2023-07-02T09:02:13.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c5194700 1 -- 172.21.15.103:0/711785203 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f03b0009e30 con 0x7f03c0132f50 2023-07-02T09:02:13.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c5194700 1 -- 172.21.15.103:0/711785203 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f03b000e360 con 0x7f03c0132f50 2023-07-02T09:02:13.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c83fa700 1 -- 172.21.15.103:0/711785203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c0132f50 msgr2=0x7f03c0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:13.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c83fa700 1 --2- 172.21.15.103:0/711785203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c0132f50 0x7f03c0133360 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c83fa700 1 -- 172.21.15.103:0/711785203 shutdown_connections 2023-07-02T09:02:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c83fa700 1 --2- 172.21.15.103:0/711785203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c0132f50 0x7f03c0133360 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c83fa700 1 -- 172.21.15.103:0/711785203 >> 172.21.15.103:0/711785203 conn(0x7f03c00a2080 msgr2=0x7f03c00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c83fa700 1 -- 172.21.15.103:0/711785203 shutdown_connections 2023-07-02T09:02:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.126+0000 7f03c83fa700 1 -- 172.21.15.103:0/711785203 wait complete. 2023-07-02T09:02:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.127+0000 7f03c83fa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.128+0000 7f03c83fa700 1 Processor -- start 2023-07-02T09:02:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.128+0000 7f03c83fa700 1 -- start start 2023-07-02T09:02:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.128+0000 7f03c83fa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c009c580 0x7f03c012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.128+0000 7f03c83fa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c006f9d0 con 0x7f03c009c580 2023-07-02T09:02:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.128+0000 7f03c6196700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c009c580 0x7f03c012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.128+0000 7f03c6196700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c009c580 0x7f03c012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42052/0 (socket says 172.21.15.103:42052) 2023-07-02T09:02:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.128+0000 7f03c6196700 1 -- 172.21.15.103:0/901980511 learned_addr learned my addr 172.21.15.103:0/901980511 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.129+0000 7f03c6196700 1 -- 172.21.15.103:0/901980511 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c0133a00 con 0x7f03c009c580 2023-07-02T09:02:13.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.129+0000 7f03c6196700 1 --2- 172.21.15.103:0/901980511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c009c580 0x7f03c012b7b0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:13.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.129+0000 7f03b6ffd700 1 -- 172.21.15.103:0/901980511 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f03b0013530 con 0x7f03c009c580 2023-07-02T09:02:13.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.129+0000 7f03b6ffd700 1 -- 172.21.15.103:0/901980511 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f03b0013690 con 0x7f03c009c580 2023-07-02T09:02:13.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.129+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c006d160 con 0x7f03c009c580 2023-07-02T09:02:13.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.129+0000 7f03b6ffd700 1 -- 172.21.15.103:0/901980511 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f03b0013930 con 0x7f03c009c580 2023-07-02T09:02:13.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.129+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c006da70 con 0x7f03c009c580 2023-07-02T09:02:13.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.130+0000 7f03b6ffd700 1 -- 172.21.15.103:0/901980511 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f03b0013ae0 con 0x7f03c009c580 2023-07-02T09:02:13.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.130+0000 7f03b6ffd700 1 --2- 172.21.15.103:0/901980511 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03a40377c0 0x7f03a4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:13.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.130+0000 7f03b6ffd700 1 -- 172.21.15.103:0/901980511 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f03b0018040 con 0x7f03c009c580 2023-07-02T09:02:13.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.130+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c0061ea0 con 0x7f03c009c580 2023-07-02T09:02:13.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.130+0000 7f03c5995700 1 --2- 172.21.15.103:0/901980511 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03a40377c0 0x7f03a4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:13.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.131+0000 7f03c5995700 1 --2- 172.21.15.103:0/901980511 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03a40377c0 0x7f03a4039cd0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:13.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.135+0000 7f03b6ffd700 1 -- 172.21.15.103:0/901980511 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f03b0018dd0 con 0x7f03c009c580 2023-07-02T09:02:13.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.323+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f03c00725b0 con 0x7f03c009c580 2023-07-02T09:02:13.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.323+0000 7f03b6ffd700 1 -- 172.21.15.103:0/901980511 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f03b00189f0 con 0x7f03c009c580 2023-07-02T09:02:13.324 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:13.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.325+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03a40377c0 msgr2=0x7f03a4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:13.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.325+0000 7f03c83fa700 1 --2- 172.21.15.103:0/901980511 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03a40377c0 0x7f03a4039cd0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:13.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.325+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c009c580 msgr2=0x7f03c012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:13.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.325+0000 7f03c83fa700 1 --2- 172.21.15.103:0/901980511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c009c580 0x7f03c012b7b0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:13.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.326+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 shutdown_connections 2023-07-02T09:02:13.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.326+0000 7f03c83fa700 1 --2- 172.21.15.103:0/901980511 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f03a40377c0 0x7f03a4039cd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:13.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.326+0000 7f03c83fa700 1 --2- 172.21.15.103:0/901980511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03c009c580 0x7f03c012b7b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:13.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.326+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 >> 172.21.15.103:0/901980511 conn(0x7f03c00a2080 msgr2=0x7f03c01babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:13.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.326+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 shutdown_connections 2023-07-02T09:02:13.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:13.326+0000 7f03c83fa700 1 -- 172.21.15.103:0/901980511 wait complete. 2023-07-02T09:02:13.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:14.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:14.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~149s 2023-07-02T09:02:14.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.479+0000 7fd80af80700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:14.484 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.484+0000 7fd80af80700 1 Processor -- start 2023-07-02T09:02:14.484 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.484+0000 7fd80af80700 1 -- start start 2023-07-02T09:02:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.484+0000 7fd80af80700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd804132030 0x7fd804134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd80af80700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80406df60 con 0x7fd80412f250 2023-07-02T09:02:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd80af80700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd80409a710 con 0x7fd804132030 2023-07-02T09:02:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd803fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd804132030 0x7fd804134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd808d1c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd80412f250 0x7fd80412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58226/0 (socket says 172.21.15.103:58226) 2023-07-02T09:02:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd808d1c700 1 -- 172.21.15.103:0/3509272095 learned_addr learned my addr 172.21.15.103:0/3509272095 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:14.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd803fff700 1 -- 172.21.15.103:0/3509272095 >> v1:172.21.15.103:6789/0 conn(0x7fd80412f250 legacy=0x7fd80412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:14.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd803fff700 1 -- 172.21.15.103:0/3509272095 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80412ed20 con 0x7fd804132030 2023-07-02T09:02:14.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd8037fe700 1 -- 172.21.15.103:0/3509272095 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010929061 0 0) 0x7fd80406df60 con 0x7fd80412f250 2023-07-02T09:02:14.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd803fff700 1 --2- 172.21.15.103:0/3509272095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd804132030 0x7fd804134450 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7aad3dcc1f3b00 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:14.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd8037fe700 1 -- 172.21.15.103:0/3509272095 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd7f0009cd0 con 0x7fd804132030 2023-07-02T09:02:14.487 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd8037fe700 1 -- 172.21.15.103:0/3509272095 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd7f0009e30 con 0x7fd804132030 2023-07-02T09:02:14.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.485+0000 7fd8037fe700 1 -- 172.21.15.103:0/3509272095 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd7f000e360 con 0x7fd804132030 2023-07-02T09:02:14.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.486+0000 7fd80af80700 1 -- 172.21.15.103:0/3509272095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd804132030 msgr2=0x7fd804134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:14.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.486+0000 7fd80af80700 1 --2- 172.21.15.103:0/3509272095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd804132030 0x7fd804134450 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:14.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.486+0000 7fd80af80700 1 -- 172.21.15.103:0/3509272095 shutdown_connections 2023-07-02T09:02:14.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.486+0000 7fd80af80700 1 --2- 172.21.15.103:0/3509272095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd804132030 0x7fd804134450 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:14.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.486+0000 7fd80af80700 1 -- 172.21.15.103:0/3509272095 >> 172.21.15.103:0/3509272095 conn(0x7fd80409d810 msgr2=0x7fd80409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:14.488 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.486+0000 7fd80af80700 1 -- 172.21.15.103:0/3509272095 shutdown_connections 2023-07-02T09:02:14.489 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.486+0000 7fd80af80700 1 -- 172.21.15.103:0/3509272095 wait complete. 2023-07-02T09:02:14.489 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.486+0000 7fd80af80700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:14.490 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.487+0000 7fd80af80700 1 Processor -- start 2023-07-02T09:02:14.490 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.487+0000 7fd80af80700 1 -- start start 2023-07-02T09:02:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.488+0000 7fd80af80700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd80412f250 0x7fd80406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.488+0000 7fd80af80700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd804127320 con 0x7fd80412f250 2023-07-02T09:02:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.488+0000 7fd808d1c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd80412f250 0x7fd80406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.488+0000 7fd808d1c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd80412f250 0x7fd80406ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36364/0 (socket says 172.21.15.103:36364) 2023-07-02T09:02:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.488+0000 7fd808d1c700 1 -- 172.21.15.103:0/1601374887 learned_addr learned my addr 172.21.15.103:0/1601374887 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.488+0000 7fd808d1c700 1 -- 172.21.15.103:0/1601374887 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8040707f0 con 0x7fd80412f250 2023-07-02T09:02:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.488+0000 7fd808d1c700 1 --2- 172.21.15.103:0/1601374887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd80412f250 0x7fd80406ed10 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.489+0000 7fd8017fa700 1 -- 172.21.15.103:0/1601374887 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd7f8004b80 con 0x7fd80412f250 2023-07-02T09:02:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.489+0000 7fd8017fa700 1 -- 172.21.15.103:0/1601374887 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd7f8004ce0 con 0x7fd80412f250 2023-07-02T09:02:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.489+0000 7fd8017fa700 1 -- 172.21.15.103:0/1601374887 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd7f800c4f0 con 0x7fd80412f250 2023-07-02T09:02:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.489+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd804071d80 con 0x7fd80412f250 2023-07-02T09:02:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.489+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8041301e0 con 0x7fd80412f250 2023-07-02T09:02:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.489+0000 7fd8017fa700 1 -- 172.21.15.103:0/1601374887 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd7f800c650 con 0x7fd80412f250 2023-07-02T09:02:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.489+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd804071bb0 con 0x7fd80412f250 2023-07-02T09:02:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.490+0000 7fd8017fa700 1 --2- 172.21.15.103:0/1601374887 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd7e80377c0 0x7fd7e8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:14.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.490+0000 7fd8017fa700 1 -- 172.21.15.103:0/1601374887 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd7f80468e0 con 0x7fd80412f250 2023-07-02T09:02:14.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.490+0000 7fd803fff700 1 --2- 172.21.15.103:0/1601374887 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd7e80377c0 0x7fd7e8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:14.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.491+0000 7fd803fff700 1 --2- 172.21.15.103:0/1601374887 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd7e80377c0 0x7fd7e8039cd0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:14.494 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.494+0000 7fd8017fa700 1 -- 172.21.15.103:0/1601374887 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd7f8009c80 con 0x7fd80412f250 2023-07-02T09:02:14.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.686+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd804061ea0 con 0x7fd80412f250 2023-07-02T09:02:14.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.686+0000 7fd8017fa700 1 -- 172.21.15.103:0/1601374887 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd7f80098a0 con 0x7fd80412f250 2023-07-02T09:02:14.686 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:14.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd7e80377c0 msgr2=0x7fd7e8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 --2- 172.21.15.103:0/1601374887 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd7e80377c0 0x7fd7e8039cd0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd80412f250 msgr2=0x7fd80406ed10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 --2- 172.21.15.103:0/1601374887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd80412f250 0x7fd80406ed10 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 shutdown_connections 2023-07-02T09:02:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 --2- 172.21.15.103:0/1601374887 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd7e80377c0 0x7fd7e8039cd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 --2- 172.21.15.103:0/1601374887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd80412f250 0x7fd80406ed10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:14.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 >> 172.21.15.103:0/1601374887 conn(0x7fd80409d810 msgr2=0x7fd804128990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:14.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 shutdown_connections 2023-07-02T09:02:14.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:14.688+0000 7fd80af80700 1 -- 172.21.15.103:0/1601374887 wait complete. 2023-07-02T09:02:14.701 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:15.702 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:15.846 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.842+0000 7fcd1b9eb700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:15.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.847+0000 7fcd1b9eb700 1 Processor -- start 2023-07-02T09:02:15.847 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.847+0000 7fcd1b9eb700 1 -- start start 2023-07-02T09:02:15.848 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd1b9eb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd1409a220 0x7fcd141348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd1b9eb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1409cea0 con 0x7fcd1406d6e0 2023-07-02T09:02:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd1b9eb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1406d490 con 0x7fcd1409a220 2023-07-02T09:02:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd18f86700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd1409a220 0x7fcd141348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd19787700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fcd1406d6e0 0x7fcd14099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58228/0 (socket says 172.21.15.103:58228) 2023-07-02T09:02:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd19787700 1 -- 172.21.15.103:0/3137617558 learned_addr learned my addr 172.21.15.103:0/3137617558 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd18f86700 1 -- 172.21.15.103:0/3137617558 >> v1:172.21.15.103:6789/0 conn(0x7fcd1406d6e0 legacy=0x7fcd14099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd18f86700 1 -- 172.21.15.103:0/3137617558 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1406dc80 con 0x7fcd1409a220 2023-07-02T09:02:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd18f86700 1 --2- 172.21.15.103:0/3137617558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1409a220 0x7fcd141348e0 crc :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb6ebda31947093 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd0bfff700 1 -- 172.21.15.103:0/3137617558 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fcd04009cd0 con 0x7fcd1409a220 2023-07-02T09:02:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd0bfff700 1 -- 172.21.15.103:0/3137617558 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fcd04009e30 con 0x7fcd1409a220 2023-07-02T09:02:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.848+0000 7fcd0bfff700 1 -- 172.21.15.103:0/3137617558 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fcd0400e360 con 0x7fcd1409a220 2023-07-02T09:02:15.851 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.849+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3137617558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1409a220 msgr2=0x7fcd141348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.849+0000 7fcd1b9eb700 1 --2- 172.21.15.103:0/3137617558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1409a220 0x7fcd141348e0 crc :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.849+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3137617558 shutdown_connections 2023-07-02T09:02:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.849+0000 7fcd1b9eb700 1 --2- 172.21.15.103:0/3137617558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1409a220 0x7fcd141348e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.849+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3137617558 >> 172.21.15.103:0/3137617558 conn(0x7fcd1409d810 msgr2=0x7fcd1409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.849+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3137617558 shutdown_connections 2023-07-02T09:02:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.849+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3137617558 wait complete. 2023-07-02T09:02:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.849+0000 7fcd1b9eb700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.850+0000 7fcd1b9eb700 1 Processor -- start 2023-07-02T09:02:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.850+0000 7fcd1b9eb700 1 -- start start 2023-07-02T09:02:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.851+0000 7fcd1b9eb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1406d6e0 0x7fcd141c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.851+0000 7fcd1b9eb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1411eaf0 con 0x7fcd1406d6e0 2023-07-02T09:02:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.851+0000 7fcd19787700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1406d6e0 0x7fcd141c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.851+0000 7fcd19787700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1406d6e0 0x7fcd141c4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36388/0 (socket says 172.21.15.103:36388) 2023-07-02T09:02:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.851+0000 7fcd19787700 1 -- 172.21.15.103:0/3228319742 learned_addr learned my addr 172.21.15.103:0/3228319742 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.851+0000 7fcd19787700 1 -- 172.21.15.103:0/3228319742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1411ac30 con 0x7fcd1406d6e0 2023-07-02T09:02:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.852+0000 7fcd19787700 1 --2- 172.21.15.103:0/3228319742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1406d6e0 0x7fcd141c4b90 crc :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.852+0000 7fcd09ffb700 1 -- 172.21.15.103:0/3228319742 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fcd10005a60 con 0x7fcd1406d6e0 2023-07-02T09:02:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.852+0000 7fcd09ffb700 1 -- 172.21.15.103:0/3228319742 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fcd10005bc0 con 0x7fcd1406d6e0 2023-07-02T09:02:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.852+0000 7fcd09ffb700 1 -- 172.21.15.103:0/3228319742 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fcd10005e60 con 0x7fcd1406d6e0 2023-07-02T09:02:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.852+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1411d150 con 0x7fcd1406d6e0 2023-07-02T09:02:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.852+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1411c4e0 con 0x7fcd1406d6e0 2023-07-02T09:02:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.853+0000 7fcd09ffb700 1 -- 172.21.15.103:0/3228319742 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fcd1000c7b0 con 0x7fcd1406d6e0 2023-07-02T09:02:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.853+0000 7fcd09ffb700 1 --2- 172.21.15.103:0/3228319742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fccf80377c0 0x7fccf8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.853+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1412fca0 con 0x7fcd1406d6e0 2023-07-02T09:02:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.853+0000 7fcd09ffb700 1 -- 172.21.15.103:0/3228319742 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fcd10046b80 con 0x7fcd1406d6e0 2023-07-02T09:02:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.853+0000 7fcd18f86700 1 --2- 172.21.15.103:0/3228319742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fccf80377c0 0x7fccf8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.854+0000 7fcd18f86700 1 --2- 172.21.15.103:0/3228319742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fccf80377c0 0x7fccf8039cd0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:15.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:15.858+0000 7fcd09ffb700 1 -- 172.21.15.103:0/3228319742 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fcd1000d150 con 0x7fcd1406d6e0 2023-07-02T09:02:16.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.046+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fcd14061ea0 con 0x7fcd1406d6e0 2023-07-02T09:02:16.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.046+0000 7fcd09ffb700 1 -- 172.21.15.103:0/3228319742 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fcd1000a4a0 con 0x7fcd1406d6e0 2023-07-02T09:02:16.047 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fccf80377c0 msgr2=0x7fccf8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 --2- 172.21.15.103:0/3228319742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fccf80377c0 0x7fccf8039cd0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1406d6e0 msgr2=0x7fcd141c4b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 --2- 172.21.15.103:0/3228319742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1406d6e0 0x7fcd141c4b90 crc :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 shutdown_connections 2023-07-02T09:02:16.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 --2- 172.21.15.103:0/3228319742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fccf80377c0 0x7fccf8039cd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:16.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 --2- 172.21.15.103:0/3228319742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd1406d6e0 0x7fcd141c4b90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:16.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 >> 172.21.15.103:0/3228319742 conn(0x7fcd1409d810 msgr2=0x7fcd141b75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:16.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 shutdown_connections 2023-07-02T09:02:16.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:16.048+0000 7fcd1b9eb700 1 -- 172.21.15.103:0/3228319742 wait complete. 2023-07-02T09:02:16.061 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:17.062 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:17.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.201+0000 7efdc89b6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:17.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.206+0000 7efdc89b6700 1 Processor -- start 2023-07-02T09:02:17.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.206+0000 7efdc89b6700 1 -- start start 2023-07-02T09:02:17.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.207+0000 7efdc89b6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdc412f250 0x7efdc412f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:17.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.207+0000 7efdc89b6700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc406df60 con 0x7efdc4098a00 2023-07-02T09:02:17.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.207+0000 7efdc89b6700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc409a710 con 0x7efdc412f250 2023-07-02T09:02:17.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.207+0000 7efdc259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdc412f250 0x7efdc412f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:17.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.207+0000 7efdc1d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7efdc4098a00 0x7efdc4098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58238/0 (socket says 172.21.15.103:58238) 2023-07-02T09:02:17.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.207+0000 7efdc259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdc412f250 0x7efdc412f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36390/0 (socket says 172.21.15.103:36390) 2023-07-02T09:02:17.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.207+0000 7efdc1d9b700 1 -- 172.21.15.103:0/3376313592 learned_addr learned my addr 172.21.15.103:0/3376313592 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:17.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.208+0000 7efdc259c700 1 -- 172.21.15.103:0/3376313592 >> v1:172.21.15.103:6789/0 conn(0x7efdc4098a00 legacy=0x7efdc4098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:17.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.208+0000 7efdc259c700 1 -- 172.21.15.103:0/3376313592 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc412ed20 con 0x7efdc412f250 2023-07-02T09:02:17.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.208+0000 7efdc159a700 1 -- 172.21.15.103:0/3376313592 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447024221 0 0) 0x7efdc406df60 con 0x7efdc4098a00 2023-07-02T09:02:17.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.208+0000 7efdc259c700 1 --2- 172.21.15.103:0/3376313592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc412f660 crc :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5fec13ff88b6af7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:17.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.208+0000 7efdc159a700 1 -- 172.21.15.103:0/3376313592 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdb800d450 con 0x7efdc412f250 2023-07-02T09:02:17.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.208+0000 7efdc159a700 1 -- 172.21.15.103:0/3376313592 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efdb800d5b0 con 0x7efdc412f250 2023-07-02T09:02:17.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.208+0000 7efdc159a700 1 -- 172.21.15.103:0/3376313592 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdb800d850 con 0x7efdc412f250 2023-07-02T09:02:17.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.209+0000 7efdc89b6700 1 -- 172.21.15.103:0/3376313592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 msgr2=0x7efdc412f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:17.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.209+0000 7efdc89b6700 1 --2- 172.21.15.103:0/3376313592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc412f660 crc :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:17.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.209+0000 7efdc89b6700 1 -- 172.21.15.103:0/3376313592 shutdown_connections 2023-07-02T09:02:17.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.209+0000 7efdc89b6700 1 --2- 172.21.15.103:0/3376313592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc412f250 0x7efdc412f660 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:17.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.209+0000 7efdc89b6700 1 -- 172.21.15.103:0/3376313592 >> 172.21.15.103:0/3376313592 conn(0x7efdc409d810 msgr2=0x7efdc409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:17.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.209+0000 7efdc89b6700 1 -- 172.21.15.103:0/3376313592 shutdown_connections 2023-07-02T09:02:17.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.209+0000 7efdc89b6700 1 -- 172.21.15.103:0/3376313592 wait complete. 2023-07-02T09:02:17.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.209+0000 7efdc89b6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:17.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.210+0000 7efdc89b6700 1 Processor -- start 2023-07-02T09:02:17.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.210+0000 7efdc89b6700 1 -- start start 2023-07-02T09:02:17.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc89b6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4098a00 0x7efdc4117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:17.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc89b6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc411a390 con 0x7efdc4098a00 2023-07-02T09:02:17.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4098a00 0x7efdc4117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:17.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4098a00 0x7efdc4117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36394/0 (socket says 172.21.15.103:36394) 2023-07-02T09:02:17.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc259c700 1 -- 172.21.15.103:0/3060045681 learned_addr learned my addr 172.21.15.103:0/3060045681 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:17.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc259c700 1 -- 172.21.15.103:0/3060045681 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc4099b60 con 0x7efdc4098a00 2023-07-02T09:02:17.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc259c700 1 --2- 172.21.15.103:0/3060045681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4098a00 0x7efdc4117360 crc :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:17.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdb2ffd700 1 -- 172.21.15.103:0/3060045681 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdb800be50 con 0x7efdc4098a00 2023-07-02T09:02:17.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdb2ffd700 1 -- 172.21.15.103:0/3060045681 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7efdb800c990 con 0x7efdc4098a00 2023-07-02T09:02:17.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc41196f0 con 0x7efdc4098a00 2023-07-02T09:02:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdb2ffd700 1 -- 172.21.15.103:0/3060045681 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7efdb800cc30 con 0x7efdc4098a00 2023-07-02T09:02:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.211+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc4119db0 con 0x7efdc4098a00 2023-07-02T09:02:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.212+0000 7efdb2ffd700 1 -- 172.21.15.103:0/3060045681 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7efdb8015040 con 0x7efdc4098a00 2023-07-02T09:02:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.212+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc4061ea0 con 0x7efdc4098a00 2023-07-02T09:02:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.212+0000 7efdb2ffd700 1 --2- 172.21.15.103:0/3060045681 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda40377c0 0x7efda4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.212+0000 7efdb2ffd700 1 -- 172.21.15.103:0/3060045681 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7efdb8012040 con 0x7efdc4098a00 2023-07-02T09:02:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.213+0000 7efdc1d9b700 1 --2- 172.21.15.103:0/3060045681 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda40377c0 0x7efda4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:17.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.213+0000 7efdc1d9b700 1 --2- 172.21.15.103:0/3060045681 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda40377c0 0x7efda4039cd0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:17.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.217+0000 7efdb2ffd700 1 -- 172.21.15.103:0/3060045681 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7efdb8013a90 con 0x7efdc4098a00 2023-07-02T09:02:17.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.408+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efdc4099b60 con 0x7efdc4098a00 2023-07-02T09:02:17.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.408+0000 7efdb2ffd700 1 -- 172.21.15.103:0/3060045681 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7efdc4099b60 con 0x7efdc4098a00 2023-07-02T09:02:17.409 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:17.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.410+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda40377c0 msgr2=0x7efda4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:17.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.410+0000 7efdc89b6700 1 --2- 172.21.15.103:0/3060045681 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda40377c0 0x7efda4039cd0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:17.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.410+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4098a00 msgr2=0x7efdc4117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:17.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.410+0000 7efdc89b6700 1 --2- 172.21.15.103:0/3060045681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4098a00 0x7efdc4117360 crc :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:17.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.410+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 shutdown_connections 2023-07-02T09:02:17.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.410+0000 7efdc89b6700 1 --2- 172.21.15.103:0/3060045681 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7efda40377c0 0x7efda4039cd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:17.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.410+0000 7efdc89b6700 1 --2- 172.21.15.103:0/3060045681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc4098a00 0x7efdc4117360 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:17.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.410+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 >> 172.21.15.103:0/3060045681 conn(0x7efdc409d810 msgr2=0x7efdc40a0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:17.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.411+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 shutdown_connections 2023-07-02T09:02:17.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:17.411+0000 7efdc89b6700 1 -- 172.21.15.103:0/3060045681 wait complete. 2023-07-02T09:02:17.423 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:18.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:18.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.562+0000 7fef319fb700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:18.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.568+0000 7fef319fb700 1 Processor -- start 2023-07-02T09:02:18.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.568+0000 7fef319fb700 1 -- start start 2023-07-02T09:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.568+0000 7fef319fb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fef2c097520 0x7fef2c097930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.568+0000 7fef319fb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c071cc0 con 0x7fef2c130570 2023-07-02T09:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.568+0000 7fef319fb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2c071270 con 0x7fef2c097520 2023-07-02T09:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef2affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fef2c097520 0x7fef2c097930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef2a7fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fef2c130570 0x7fef2c130980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58248/0 (socket says 172.21.15.103:58248) 2023-07-02T09:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef2affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fef2c097520 0x7fef2c097930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36408/0 (socket says 172.21.15.103:36408) 2023-07-02T09:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef2a7fc700 1 -- 172.21.15.103:0/3470635790 learned_addr learned my addr 172.21.15.103:0/3470635790 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef2affd700 1 -- 172.21.15.103:0/3470635790 >> v1:172.21.15.103:6789/0 conn(0x7fef2c130570 legacy=0x7fef2c130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef29ffb700 1 -- 172.21.15.103:0/3470635790 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191814611 0 0) 0x7fef2c071cc0 con 0x7fef2c130570 2023-07-02T09:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef2affd700 1 -- 172.21.15.103:0/3470635790 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c09d1c0 con 0x7fef2c097520 2023-07-02T09:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef2affd700 1 --2- 172.21.15.103:0/3470635790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c097930 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a29b1b7e6347c4 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef29ffb700 1 -- 172.21.15.103:0/3470635790 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fef1c009cd0 con 0x7fef2c097520 2023-07-02T09:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef29ffb700 1 -- 172.21.15.103:0/3470635790 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fef1c009e30 con 0x7fef2c097520 2023-07-02T09:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.569+0000 7fef29ffb700 1 -- 172.21.15.103:0/3470635790 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fef1c00e360 con 0x7fef2c097520 2023-07-02T09:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.570+0000 7fef319fb700 1 -- 172.21.15.103:0/3470635790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 msgr2=0x7fef2c097930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.570+0000 7fef319fb700 1 --2- 172.21.15.103:0/3470635790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c097930 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.570+0000 7fef319fb700 1 -- 172.21.15.103:0/3470635790 shutdown_connections 2023-07-02T09:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.570+0000 7fef319fb700 1 --2- 172.21.15.103:0/3470635790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c097930 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.570+0000 7fef319fb700 1 -- 172.21.15.103:0/3470635790 >> 172.21.15.103:0/3470635790 conn(0x7fef2c0a1030 msgr2=0x7fef2c0a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.570+0000 7fef319fb700 1 -- 172.21.15.103:0/3470635790 shutdown_connections 2023-07-02T09:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.570+0000 7fef319fb700 1 -- 172.21.15.103:0/3470635790 wait complete. 2023-07-02T09:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.570+0000 7fef319fb700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.571+0000 7fef319fb700 1 Processor -- start 2023-07-02T09:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.571+0000 7fef319fb700 1 -- start start 2023-07-02T09:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.572+0000 7fef319fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c127ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.572+0000 7fef319fb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c127a20 con 0x7fef2c097520 2023-07-02T09:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.572+0000 7fef2affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c127ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.572+0000 7fef2affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c127ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36422/0 (socket says 172.21.15.103:36422) 2023-07-02T09:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.572+0000 7fef2affd700 1 -- 172.21.15.103:0/3943997619 learned_addr learned my addr 172.21.15.103:0/3943997619 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.572+0000 7fef2affd700 1 -- 172.21.15.103:0/3943997619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c129600 con 0x7fef2c097520 2023-07-02T09:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.573+0000 7fef2affd700 1 --2- 172.21.15.103:0/3943997619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c127ff0 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.573+0000 7fef13fff700 1 -- 172.21.15.103:0/3943997619 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fef1c014540 con 0x7fef2c097520 2023-07-02T09:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.573+0000 7fef13fff700 1 -- 172.21.15.103:0/3943997619 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fef1c0146a0 con 0x7fef2c097520 2023-07-02T09:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.573+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c127c50 con 0x7fef2c097520 2023-07-02T09:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.573+0000 7fef13fff700 1 -- 172.21.15.103:0/3943997619 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fef1c014940 con 0x7fef2c097520 2023-07-02T09:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.573+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c12a570 con 0x7fef2c097520 2023-07-02T09:02:18.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.574+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c127e40 con 0x7fef2c097520 2023-07-02T09:02:18.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.574+0000 7fef13fff700 1 -- 172.21.15.103:0/3943997619 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fef1c014af0 con 0x7fef2c097520 2023-07-02T09:02:18.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.574+0000 7fef13fff700 1 --2- 172.21.15.103:0/3943997619 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fef180377c0 0x7fef18039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:18.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.574+0000 7fef13fff700 1 -- 172.21.15.103:0/3943997619 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fef1c048450 con 0x7fef2c097520 2023-07-02T09:02:18.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.574+0000 7fef2a7fc700 1 --2- 172.21.15.103:0/3943997619 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fef180377c0 0x7fef18039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:18.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.575+0000 7fef2a7fc700 1 --2- 172.21.15.103:0/3943997619 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fef180377c0 0x7fef18039cd0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:18.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.579+0000 7fef13fff700 1 -- 172.21.15.103:0/3943997619 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fef1c00fe40 con 0x7fef2c097520 2023-07-02T09:02:18.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.770+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fef2c061ea0 con 0x7fef2c097520 2023-07-02T09:02:18.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.771+0000 7fef13fff700 1 -- 172.21.15.103:0/3943997619 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fef1c011320 con 0x7fef2c097520 2023-07-02T09:02:18.771 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:18.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fef180377c0 msgr2=0x7fef18039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:18.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 --2- 172.21.15.103:0/3943997619 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fef180377c0 0x7fef18039cd0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:18.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 msgr2=0x7fef2c127ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:18.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 --2- 172.21.15.103:0/3943997619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c127ff0 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:18.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 shutdown_connections 2023-07-02T09:02:18.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 --2- 172.21.15.103:0/3943997619 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fef180377c0 0x7fef18039cd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:18.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 --2- 172.21.15.103:0/3943997619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2c097520 0x7fef2c127ff0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:18.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 >> 172.21.15.103:0/3943997619 conn(0x7fef2c0a1030 msgr2=0x7fef2c12e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:18.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 shutdown_connections 2023-07-02T09:02:18.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:18.773+0000 7fef319fb700 1 -- 172.21.15.103:0/3943997619 wait complete. 2023-07-02T09:02:18.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:19.787 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:19.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.926+0000 7ff14023f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:19.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.931+0000 7ff14023f700 1 Processor -- start 2023-07-02T09:02:19.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.931+0000 7ff14023f700 1 -- start start 2023-07-02T09:02:19.932 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff14023f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff13812f250 0x7ff13812f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:19.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff14023f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13806df60 con 0x7ff138098a00 2023-07-02T09:02:19.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff14023f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13809a710 con 0x7ff13812f250 2023-07-02T09:02:19.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff13dfdb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff13812f250 0x7ff13812f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:19.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff13d7da700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff138098a00 0x7ff138098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58256/0 (socket says 172.21.15.103:58256) 2023-07-02T09:02:19.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff13d7da700 1 -- 172.21.15.103:0/1492331714 learned_addr learned my addr 172.21.15.103:0/1492331714 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:19.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff13dfdb700 1 -- 172.21.15.103:0/1492331714 >> v1:172.21.15.103:6789/0 conn(0x7ff138098a00 legacy=0x7ff138098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:19.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff13dfdb700 1 -- 172.21.15.103:0/1492331714 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13812ed20 con 0x7ff13812f250 2023-07-02T09:02:19.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff13cfd9700 1 -- 172.21.15.103:0/1492331714 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395379203 0 0) 0x7ff13806df60 con 0x7ff138098a00 2023-07-02T09:02:19.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff13dfdb700 1 --2- 172.21.15.103:0/1492331714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13812f250 0x7ff13812f660 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b3ab13807cd443 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:19.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.932+0000 7ff13cfd9700 1 -- 172.21.15.103:0/1492331714 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff12800ac60 con 0x7ff13812f250 2023-07-02T09:02:19.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff13cfd9700 1 -- 172.21.15.103:0/1492331714 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff12800adc0 con 0x7ff13812f250 2023-07-02T09:02:19.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff13cfd9700 1 -- 172.21.15.103:0/1492331714 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff12800f470 con 0x7ff13812f250 2023-07-02T09:02:19.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff14023f700 1 -- 172.21.15.103:0/1492331714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13812f250 msgr2=0x7ff13812f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:19.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff14023f700 1 --2- 172.21.15.103:0/1492331714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13812f250 0x7ff13812f660 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:19.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff14023f700 1 -- 172.21.15.103:0/1492331714 shutdown_connections 2023-07-02T09:02:19.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff14023f700 1 --2- 172.21.15.103:0/1492331714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13812f250 0x7ff13812f660 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:19.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff14023f700 1 -- 172.21.15.103:0/1492331714 >> 172.21.15.103:0/1492331714 conn(0x7ff13809d810 msgr2=0x7ff13809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:19.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff14023f700 1 -- 172.21.15.103:0/1492331714 shutdown_connections 2023-07-02T09:02:19.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.933+0000 7ff14023f700 1 -- 172.21.15.103:0/1492331714 wait complete. 2023-07-02T09:02:19.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.934+0000 7ff14023f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:19.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.934+0000 7ff14023f700 1 Processor -- start 2023-07-02T09:02:19.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.934+0000 7ff14023f700 1 -- start start 2023-07-02T09:02:19.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.935+0000 7ff14023f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff138098a00 0x7ff138117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:19.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.935+0000 7ff14023f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138118750 con 0x7ff138098a00 2023-07-02T09:02:19.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.935+0000 7ff13dfdb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff138098a00 0x7ff138117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:19.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.935+0000 7ff13dfdb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff138098a00 0x7ff138117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36444/0 (socket says 172.21.15.103:36444) 2023-07-02T09:02:19.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.935+0000 7ff13dfdb700 1 -- 172.21.15.103:0/2950092767 learned_addr learned my addr 172.21.15.103:0/2950092767 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:19.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.935+0000 7ff13dfdb700 1 -- 172.21.15.103:0/2950092767 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13811a450 con 0x7ff138098a00 2023-07-02T09:02:19.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.936+0000 7ff13dfdb700 1 --2- 172.21.15.103:0/2950092767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff138098a00 0x7ff138117360 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:19.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.936+0000 7ff12e7fc700 1 -- 172.21.15.103:0/2950092767 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff128017040 con 0x7ff138098a00 2023-07-02T09:02:19.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.936+0000 7ff12e7fc700 1 -- 172.21.15.103:0/2950092767 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff12800fe00 con 0x7ff138098a00 2023-07-02T09:02:19.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.936+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff138118e20 con 0x7ff138098a00 2023-07-02T09:02:19.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.936+0000 7ff12e7fc700 1 -- 172.21.15.103:0/2950092767 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff128011550 con 0x7ff138098a00 2023-07-02T09:02:19.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.936+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff138119e70 con 0x7ff138098a00 2023-07-02T09:02:19.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.937+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff138061ea0 con 0x7ff138098a00 2023-07-02T09:02:19.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.937+0000 7ff12e7fc700 1 -- 172.21.15.103:0/2950092767 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ff128011700 con 0x7ff138098a00 2023-07-02T09:02:19.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.937+0000 7ff12e7fc700 1 --2- 172.21.15.103:0/2950092767 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff11c0377c0 0x7ff11c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:19.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.937+0000 7ff12e7fc700 1 -- 172.21.15.103:0/2950092767 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ff128048bf0 con 0x7ff138098a00 2023-07-02T09:02:19.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.937+0000 7ff13d7da700 1 --2- 172.21.15.103:0/2950092767 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff11c0377c0 0x7ff11c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:19.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.938+0000 7ff13d7da700 1 --2- 172.21.15.103:0/2950092767 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff11c0377c0 0x7ff11c039cd0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:19.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:19.942+0000 7ff12e7fc700 1 -- 172.21.15.103:0/2950092767 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ff12800c660 con 0x7ff138098a00 2023-07-02T09:02:19.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~154s 2023-07-02T09:02:20.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.130+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff138073ca0 con 0x7ff138098a00 2023-07-02T09:02:20.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.130+0000 7ff12e7fc700 1 -- 172.21.15.103:0/2950092767 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ff128006860 con 0x7ff138098a00 2023-07-02T09:02:20.132 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.132+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff11c0377c0 msgr2=0x7ff11c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.132+0000 7ff14023f700 1 --2- 172.21.15.103:0/2950092767 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff11c0377c0 0x7ff11c039cd0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.132+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff138098a00 msgr2=0x7ff138117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.132+0000 7ff14023f700 1 --2- 172.21.15.103:0/2950092767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff138098a00 0x7ff138117360 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.132+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 shutdown_connections 2023-07-02T09:02:20.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.132+0000 7ff14023f700 1 --2- 172.21.15.103:0/2950092767 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff11c0377c0 0x7ff11c039cd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:20.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.133+0000 7ff14023f700 1 --2- 172.21.15.103:0/2950092767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff138098a00 0x7ff138117360 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:20.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.133+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 >> 172.21.15.103:0/2950092767 conn(0x7ff13809d810 msgr2=0x7ff1380a0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:20.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.133+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 shutdown_connections 2023-07-02T09:02:20.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:20.133+0000 7ff14023f700 1 -- 172.21.15.103:0/2950092767 wait complete. 2023-07-02T09:02:20.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:21.148 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:21.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.286+0000 7fc3dc65a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:21.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.291+0000 7fc3dc65a700 1 Processor -- start 2023-07-02T09:02:21.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.291+0000 7fc3dc65a700 1 -- start start 2023-07-02T09:02:21.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.292+0000 7fc3dc65a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc3d412f250 0x7fc3d412f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:21.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.292+0000 7fc3dc65a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d406df60 con 0x7fc3d4098a00 2023-07-02T09:02:21.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.292+0000 7fc3dc65a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d409a710 con 0x7fc3d412f250 2023-07-02T09:02:21.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.292+0000 7fc3da3f6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc3d412f250 0x7fc3d412f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:21.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.292+0000 7fc3da3f6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc3d412f250 0x7fc3d412f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36454/0 (socket says 172.21.15.103:36454) 2023-07-02T09:02:21.294 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.292+0000 7fc3d9bf5700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc3d4098a00 0x7fc3d4098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58258/0 (socket says 172.21.15.103:58258) 2023-07-02T09:02:21.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.292+0000 7fc3da3f6700 1 -- 172.21.15.103:0/2255895528 learned_addr learned my addr 172.21.15.103:0/2255895528 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:21.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.293+0000 7fc3da3f6700 1 -- 172.21.15.103:0/2255895528 >> v1:172.21.15.103:6789/0 conn(0x7fc3d4098a00 legacy=0x7fc3d4098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:21.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.293+0000 7fc3da3f6700 1 -- 172.21.15.103:0/2255895528 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d412ed20 con 0x7fc3d412f250 2023-07-02T09:02:21.295 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.293+0000 7fc3da3f6700 1 --2- 172.21.15.103:0/2255895528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d412f250 0x7fc3d412f660 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc51d9f4d24f9e9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:21.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.293+0000 7fc3d93f4700 1 -- 172.21.15.103:0/2255895528 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc3c4009d90 con 0x7fc3d412f250 2023-07-02T09:02:21.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.293+0000 7fc3d93f4700 1 -- 172.21.15.103:0/2255895528 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc3c400f340 con 0x7fc3d412f250 2023-07-02T09:02:21.296 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.293+0000 7fc3d93f4700 1 -- 172.21.15.103:0/2255895528 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc3c400f5e0 con 0x7fc3d412f250 2023-07-02T09:02:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.294+0000 7fc3dc65a700 1 -- 172.21.15.103:0/2255895528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d412f250 msgr2=0x7fc3d412f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.294+0000 7fc3dc65a700 1 --2- 172.21.15.103:0/2255895528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d412f250 0x7fc3d412f660 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.294+0000 7fc3dc65a700 1 -- 172.21.15.103:0/2255895528 shutdown_connections 2023-07-02T09:02:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.294+0000 7fc3dc65a700 1 --2- 172.21.15.103:0/2255895528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d412f250 0x7fc3d412f660 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.294+0000 7fc3dc65a700 1 -- 172.21.15.103:0/2255895528 >> 172.21.15.103:0/2255895528 conn(0x7fc3d409d810 msgr2=0x7fc3d409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.294+0000 7fc3dc65a700 1 -- 172.21.15.103:0/2255895528 shutdown_connections 2023-07-02T09:02:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.294+0000 7fc3dc65a700 1 -- 172.21.15.103:0/2255895528 wait complete. 2023-07-02T09:02:21.297 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.294+0000 7fc3dc65a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:21.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.295+0000 7fc3dc65a700 1 Processor -- start 2023-07-02T09:02:21.298 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.295+0000 7fc3dc65a700 1 -- start start 2023-07-02T09:02:21.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.295+0000 7fc3dc65a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d4098a00 0x7fc3d41194d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:21.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.295+0000 7fc3dc65a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d411a9c0 con 0x7fc3d4098a00 2023-07-02T09:02:21.299 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3da3f6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d4098a00 0x7fc3d41194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:21.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3da3f6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d4098a00 0x7fc3d41194d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36460/0 (socket says 172.21.15.103:36460) 2023-07-02T09:02:21.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3da3f6700 1 -- 172.21.15.103:0/3638481745 learned_addr learned my addr 172.21.15.103:0/3638481745 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:21.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3da3f6700 1 -- 172.21.15.103:0/3638481745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d411bba0 con 0x7fc3d4098a00 2023-07-02T09:02:21.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3da3f6700 1 --2- 172.21.15.103:0/3638481745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d4098a00 0x7fc3d41194d0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:21.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3caffd700 1 -- 172.21.15.103:0/3638481745 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc3c400fdc0 con 0x7fc3d4098a00 2023-07-02T09:02:21.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3caffd700 1 -- 172.21.15.103:0/3638481745 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc3c4011420 con 0x7fc3d4098a00 2023-07-02T09:02:21.300 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d4128c90 con 0x7fc3d4098a00 2023-07-02T09:02:21.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3caffd700 1 -- 172.21.15.103:0/3638481745 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc3c40116c0 con 0x7fc3d4098a00 2023-07-02T09:02:21.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.296+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d411c740 con 0x7fc3d4098a00 2023-07-02T09:02:21.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.297+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d4061ea0 con 0x7fc3d4098a00 2023-07-02T09:02:21.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.297+0000 7fc3caffd700 1 -- 172.21.15.103:0/3638481745 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fc3c4011870 con 0x7fc3d4098a00 2023-07-02T09:02:21.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.297+0000 7fc3caffd700 1 --2- 172.21.15.103:0/3638481745 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3b80377c0 0x7fc3b8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:21.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.297+0000 7fc3caffd700 1 -- 172.21.15.103:0/3638481745 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc3c4047800 con 0x7fc3d4098a00 2023-07-02T09:02:21.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.298+0000 7fc3d9bf5700 1 --2- 172.21.15.103:0/3638481745 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3b80377c0 0x7fc3b8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:21.301 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.298+0000 7fc3d9bf5700 1 --2- 172.21.15.103:0/3638481745 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3b80377c0 0x7fc3b8039cd0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:21.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.302+0000 7fc3caffd700 1 -- 172.21.15.103:0/3638481745 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc3c4015300 con 0x7fc3d4098a00 2023-07-02T09:02:21.494 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.493+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc3d4073ca0 con 0x7fc3d4098a00 2023-07-02T09:02:21.494 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.493+0000 7fc3caffd700 1 -- 172.21.15.103:0/3638481745 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fc3c4015120 con 0x7fc3d4098a00 2023-07-02T09:02:21.495 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:21.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3b80377c0 msgr2=0x7fc3b8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:21.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 --2- 172.21.15.103:0/3638481745 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3b80377c0 0x7fc3b8039cd0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:21.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d4098a00 msgr2=0x7fc3d41194d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:21.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 --2- 172.21.15.103:0/3638481745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d4098a00 0x7fc3d41194d0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:21.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 shutdown_connections 2023-07-02T09:02:21.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 --2- 172.21.15.103:0/3638481745 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc3b80377c0 0x7fc3b8039cd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:21.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 --2- 172.21.15.103:0/3638481745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3d4098a00 0x7fc3d41194d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:21.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 >> 172.21.15.103:0/3638481745 conn(0x7fc3d409d810 msgr2=0x7fc3d412cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:21.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 shutdown_connections 2023-07-02T09:02:21.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:21.495+0000 7fc3dc65a700 1 -- 172.21.15.103:0/3638481745 wait complete. 2023-07-02T09:02:21.509 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:22.510 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:22.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.648+0000 7f58d782f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:22.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.653+0000 7f58d782f700 1 Processor -- start 2023-07-02T09:02:22.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.654+0000 7f58d782f700 1 -- start start 2023-07-02T09:02:22.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.654+0000 7f58d782f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f58d0133360 0x7f58d009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:22.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.654+0000 7f58d782f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d009e260 con 0x7f58d0132f50 2023-07-02T09:02:22.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.654+0000 7f58d782f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d00a1640 con 0x7f58d0133360 2023-07-02T09:02:22.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.654+0000 7f58d4dca700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f58d0133360 0x7f58d009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:22.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.654+0000 7f58d55cb700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f58d0132f50 0x7f58d009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58274/0 (socket says 172.21.15.103:58274) 2023-07-02T09:02:22.656 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.654+0000 7f58d55cb700 1 -- 172.21.15.103:0/974189940 learned_addr learned my addr 172.21.15.103:0/974189940 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:22.657 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.655+0000 7f58d4dca700 1 -- 172.21.15.103:0/974189940 >> v1:172.21.15.103:6789/0 conn(0x7f58d0132f50 legacy=0x7f58d009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:22.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.655+0000 7f58d4dca700 1 -- 172.21.15.103:0/974189940 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d0132a20 con 0x7f58d0133360 2023-07-02T09:02:22.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.655+0000 7f58c7fff700 1 -- 172.21.15.103:0/974189940 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632484685 0 0) 0x7f58d009e260 con 0x7f58d0132f50 2023-07-02T09:02:22.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.655+0000 7f58d4dca700 1 --2- 172.21.15.103:0/974189940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0133360 0x7f58d009cc90 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1e2f935315094a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:22.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.655+0000 7f58c7fff700 1 -- 172.21.15.103:0/974189940 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f58cc009cd0 con 0x7f58d0133360 2023-07-02T09:02:22.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.655+0000 7f58c7fff700 1 -- 172.21.15.103:0/974189940 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f58cc009e30 con 0x7f58d0133360 2023-07-02T09:02:22.658 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.655+0000 7f58c7fff700 1 -- 172.21.15.103:0/974189940 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f58cc00e360 con 0x7f58d0133360 2023-07-02T09:02:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.656+0000 7f58d782f700 1 -- 172.21.15.103:0/974189940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0133360 msgr2=0x7f58d009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.656+0000 7f58d782f700 1 --2- 172.21.15.103:0/974189940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0133360 0x7f58d009cc90 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.656+0000 7f58d782f700 1 -- 172.21.15.103:0/974189940 shutdown_connections 2023-07-02T09:02:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.656+0000 7f58d782f700 1 --2- 172.21.15.103:0/974189940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0133360 0x7f58d009cc90 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.656+0000 7f58d782f700 1 -- 172.21.15.103:0/974189940 >> 172.21.15.103:0/974189940 conn(0x7f58d00a2080 msgr2=0x7f58d00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.656+0000 7f58d782f700 1 -- 172.21.15.103:0/974189940 shutdown_connections 2023-07-02T09:02:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.656+0000 7f58d782f700 1 -- 172.21.15.103:0/974189940 wait complete. 2023-07-02T09:02:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.656+0000 7f58d782f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:22.661 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.657+0000 7f58d782f700 1 Processor -- start 2023-07-02T09:02:22.661 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.657+0000 7f58d782f700 1 -- start start 2023-07-02T09:02:22.661 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58d782f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0132f50 0x7f58d01ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:22.661 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58d782f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d006d370 con 0x7f58d0132f50 2023-07-02T09:02:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58d55cb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0132f50 0x7f58d01ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58d55cb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0132f50 0x7f58d01ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36478/0 (socket says 172.21.15.103:36478) 2023-07-02T09:02:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58d55cb700 1 -- 172.21.15.103:0/4111111729 learned_addr learned my addr 172.21.15.103:0/4111111729 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58d55cb700 1 -- 172.21.15.103:0/4111111729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d006dde0 con 0x7f58d0132f50 2023-07-02T09:02:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58d55cb700 1 --2- 172.21.15.103:0/4111111729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0132f50 0x7f58d01ba9b0 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58c5ffb700 1 -- 172.21.15.103:0/4111111729 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f58c0004b80 con 0x7f58d0132f50 2023-07-02T09:02:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58c5ffb700 1 -- 172.21.15.103:0/4111111729 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f58c0004ce0 con 0x7f58d0132f50 2023-07-02T09:02:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d01ba4f0 con 0x7f58d0132f50 2023-07-02T09:02:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.658+0000 7f58c5ffb700 1 -- 172.21.15.103:0/4111111729 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f58c000c4f0 con 0x7f58d0132f50 2023-07-02T09:02:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.659+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d0071e40 con 0x7f58d0132f50 2023-07-02T09:02:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.659+0000 7f58c5ffb700 1 -- 172.21.15.103:0/4111111729 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f58c000c650 con 0x7f58d0132f50 2023-07-02T09:02:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.660+0000 7f58c5ffb700 1 --2- 172.21.15.103:0/4111111729 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f58b40377c0 0x7f58b4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.660+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d0061ea0 con 0x7f58d0132f50 2023-07-02T09:02:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.660+0000 7f58c5ffb700 1 -- 172.21.15.103:0/4111111729 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f58c0046a30 con 0x7f58d0132f50 2023-07-02T09:02:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.660+0000 7f58d4dca700 1 --2- 172.21.15.103:0/4111111729 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f58b40377c0 0x7f58b4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.660+0000 7f58d4dca700 1 --2- 172.21.15.103:0/4111111729 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f58b40377c0 0x7f58b4039cd0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:22.665 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.664+0000 7f58c5ffb700 1 -- 172.21.15.103:0/4111111729 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f58c0009c80 con 0x7f58d0132f50 2023-07-02T09:02:22.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.856+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f58d00725b0 con 0x7f58d0132f50 2023-07-02T09:02:22.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.856+0000 7f58c5ffb700 1 -- 172.21.15.103:0/4111111729 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f58c00098a0 con 0x7f58d0132f50 2023-07-02T09:02:22.858 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:22.860 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f58b40377c0 msgr2=0x7f58b4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:22.860 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 --2- 172.21.15.103:0/4111111729 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f58b40377c0 0x7f58b4039cd0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:22.860 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0132f50 msgr2=0x7f58d01ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 --2- 172.21.15.103:0/4111111729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0132f50 0x7f58d01ba9b0 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 shutdown_connections 2023-07-02T09:02:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 --2- 172.21.15.103:0/4111111729 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f58b40377c0 0x7f58b4039cd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 --2- 172.21.15.103:0/4111111729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d0132f50 0x7f58d01ba9b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 >> 172.21.15.103:0/4111111729 conn(0x7f58d00a2080 msgr2=0x7f58d00a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 shutdown_connections 2023-07-02T09:02:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:22.859+0000 7f58d782f700 1 -- 172.21.15.103:0/4111111729 wait complete. 2023-07-02T09:02:22.873 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:23.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:24.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.012+0000 7f6dd92b9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:24.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.017+0000 7f6dd92b9700 1 Processor -- start 2023-07-02T09:02:24.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.017+0000 7f6dd92b9700 1 -- start start 2023-07-02T09:02:24.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd92b9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6dd409a220 0x7f6dd41348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:24.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd92b9700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd409cea0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd92b9700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd406d490 con 0x7f6dd409a220 2023-07-02T09:02:24.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6dd409a220 0x7f6dd41348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:24.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6dd409a220 0x7f6dd41348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55122/0 (socket says 172.21.15.103:55122) 2023-07-02T09:02:24.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd2ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6dd406d6e0 0x7f6dd4099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54480/0 (socket says 172.21.15.103:54480) 2023-07-02T09:02:24.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd27fc700 1 -- 172.21.15.103:0/557688263 learned_addr learned my addr 172.21.15.103:0/557688263 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:24.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd27fc700 1 -- 172.21.15.103:0/557688263 >> v1:172.21.15.103:6789/0 conn(0x7f6dd406d6e0 legacy=0x7f6dd4099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd27fc700 1 -- 172.21.15.103:0/557688263 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd406dc80 con 0x7f6dd409a220 2023-07-02T09:02:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd1ffb700 1 -- 172.21.15.103:0/557688263 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101904141 0 0) 0x7f6dd409cea0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd27fc700 1 --2- 172.21.15.103:0/557688263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd409a220 0x7f6dd41348e0 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7592bcd2134601c server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd1ffb700 1 -- 172.21.15.103:0/557688263 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6dc4010570 con 0x7f6dd409a220 2023-07-02T09:02:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.018+0000 7f6dd1ffb700 1 -- 172.21.15.103:0/557688263 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6dc40106d0 con 0x7f6dd409a220 2023-07-02T09:02:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.019+0000 7f6dd1ffb700 1 -- 172.21.15.103:0/557688263 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6dc4010970 con 0x7f6dd409a220 2023-07-02T09:02:24.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.019+0000 7f6dd92b9700 1 -- 172.21.15.103:0/557688263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd409a220 msgr2=0x7f6dd41348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:24.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.019+0000 7f6dd92b9700 1 --2- 172.21.15.103:0/557688263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd409a220 0x7f6dd41348e0 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:24.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.019+0000 7f6dd92b9700 1 -- 172.21.15.103:0/557688263 shutdown_connections 2023-07-02T09:02:24.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.019+0000 7f6dd92b9700 1 --2- 172.21.15.103:0/557688263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd409a220 0x7f6dd41348e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:24.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.019+0000 7f6dd92b9700 1 -- 172.21.15.103:0/557688263 >> 172.21.15.103:0/557688263 conn(0x7f6dd409d810 msgr2=0x7f6dd409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:24.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.019+0000 7f6dd92b9700 1 -- 172.21.15.103:0/557688263 shutdown_connections 2023-07-02T09:02:24.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.019+0000 7f6dd92b9700 1 -- 172.21.15.103:0/557688263 wait complete. 2023-07-02T09:02:24.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.020+0000 7f6dd92b9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:24.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.021+0000 7f6dd92b9700 1 Processor -- start 2023-07-02T09:02:24.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.021+0000 7f6dd92b9700 1 -- start start 2023-07-02T09:02:24.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd92b9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd406d6e0 0x7f6dd41c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:24.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd92b9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd4121050 con 0x7f6dd406d6e0 2023-07-02T09:02:24.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd406d6e0 0x7f6dd41c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:24.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd406d6e0 0x7f6dd41c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55128/0 (socket says 172.21.15.103:55128) 2023-07-02T09:02:24.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd2ffd700 1 -- 172.21.15.103:0/1821185388 learned_addr learned my addr 172.21.15.103:0/1821185388 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:24.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd2ffd700 1 -- 172.21.15.103:0/1821185388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd4124290 con 0x7f6dd406d6e0 2023-07-02T09:02:24.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd2ffd700 1 --2- 172.21.15.103:0/1821185388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd406d6e0 0x7f6dd41c3a00 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:24.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dbb7fe700 1 -- 172.21.15.103:0/1821185388 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6dc8004b80 con 0x7f6dd406d6e0 2023-07-02T09:02:24.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dbb7fe700 1 -- 172.21.15.103:0/1821185388 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6dc8004ce0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd4120cb0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dbb7fe700 1 -- 172.21.15.103:0/1821185388 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6dc800c550 con 0x7f6dd406d6e0 2023-07-02T09:02:24.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.022+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd4124890 con 0x7f6dd406d6e0 2023-07-02T09:02:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.023+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd4120ea0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.023+0000 7f6dbb7fe700 1 -- 172.21.15.103:0/1821185388 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6dc800c6b0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.024+0000 7f6dbb7fe700 1 --2- 172.21.15.103:0/1821185388 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6dc00377c0 0x7f6dc0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.024+0000 7f6dbb7fe700 1 -- 172.21.15.103:0/1821185388 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6dc80469e0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.024+0000 7f6dd27fc700 1 --2- 172.21.15.103:0/1821185388 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6dc00377c0 0x7f6dc0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.024+0000 7f6dd27fc700 1 --2- 172.21.15.103:0/1821185388 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6dc00377c0 0x7f6dc0039cd0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:24.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.028+0000 7f6dbb7fe700 1 -- 172.21.15.103:0/1821185388 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6dc8009a80 con 0x7f6dd406d6e0 2023-07-02T09:02:24.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.219+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6dd4061ea0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.219+0000 7f6dbb7fe700 1 -- 172.21.15.103:0/1821185388 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6dc80098a0 con 0x7f6dd406d6e0 2023-07-02T09:02:24.221 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:24.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.221+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6dc00377c0 msgr2=0x7f6dc0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:24.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.221+0000 7f6dd92b9700 1 --2- 172.21.15.103:0/1821185388 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6dc00377c0 0x7f6dc0039cd0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:24.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.221+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd406d6e0 msgr2=0x7f6dd41c3a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:24.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.221+0000 7f6dd92b9700 1 --2- 172.21.15.103:0/1821185388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd406d6e0 0x7f6dd41c3a00 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:24.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.222+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 shutdown_connections 2023-07-02T09:02:24.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.222+0000 7f6dd92b9700 1 --2- 172.21.15.103:0/1821185388 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6dc00377c0 0x7f6dc0039cd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:24.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.222+0000 7f6dd92b9700 1 --2- 172.21.15.103:0/1821185388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd406d6e0 0x7f6dd41c3a00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:24.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.222+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 >> 172.21.15.103:0/1821185388 conn(0x7f6dd409d810 msgr2=0x7f6dd4070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:24.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.222+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 shutdown_connections 2023-07-02T09:02:24.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:24.222+0000 7f6dd92b9700 1 -- 172.21.15.103:0/1821185388 wait complete. 2023-07-02T09:02:24.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:25.236 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:25.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.374+0000 7f7a5b899700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:25.380 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.379+0000 7f7a5b899700 1 Processor -- start 2023-07-02T09:02:25.380 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.379+0000 7f7a5b899700 1 -- start start 2023-07-02T09:02:25.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a5b899700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7a54099520 0x7f7a54135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:25.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a5b899700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5406f8b0 con 0x7f7a54098a00 2023-07-02T09:02:25.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a5b899700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a54072510 con 0x7f7a54099520 2023-07-02T09:02:25.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a58e34700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7a54099520 0x7f7a54135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:25.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a59635700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7a54098a00 0x7f7a54098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54484/0 (socket says 172.21.15.103:54484) 2023-07-02T09:02:25.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a58e34700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7a54099520 0x7f7a54135ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55132/0 (socket says 172.21.15.103:55132) 2023-07-02T09:02:25.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a59635700 1 -- 172.21.15.103:0/3431840035 learned_addr learned my addr 172.21.15.103:0/3431840035 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:25.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a58e34700 1 -- 172.21.15.103:0/3431840035 >> v1:172.21.15.103:6789/0 conn(0x7f7a54098a00 legacy=0x7f7a54098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:25.383 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.380+0000 7f7a58e34700 1 -- 172.21.15.103:0/3431840035 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5409aec0 con 0x7f7a54099520 2023-07-02T09:02:25.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a58e34700 1 --2- 172.21.15.103:0/3431840035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54099520 0x7f7a54135ca0 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e554b334a2367be server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:25.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a4bfff700 1 -- 172.21.15.103:0/3431840035 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a50009cd0 con 0x7f7a54099520 2023-07-02T09:02:25.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a4bfff700 1 -- 172.21.15.103:0/3431840035 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7a50009e30 con 0x7f7a54099520 2023-07-02T09:02:25.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a4bfff700 1 -- 172.21.15.103:0/3431840035 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a5000e360 con 0x7f7a54099520 2023-07-02T09:02:25.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a5b899700 1 -- 172.21.15.103:0/3431840035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54099520 msgr2=0x7f7a54135ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:25.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a5b899700 1 --2- 172.21.15.103:0/3431840035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54099520 0x7f7a54135ca0 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:25.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a5b899700 1 -- 172.21.15.103:0/3431840035 shutdown_connections 2023-07-02T09:02:25.384 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a5b899700 1 --2- 172.21.15.103:0/3431840035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54099520 0x7f7a54135ca0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:25.385 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a5b899700 1 -- 172.21.15.103:0/3431840035 >> 172.21.15.103:0/3431840035 conn(0x7f7a5409d810 msgr2=0x7f7a5409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:25.385 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a5b899700 1 -- 172.21.15.103:0/3431840035 shutdown_connections 2023-07-02T09:02:25.385 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.381+0000 7f7a5b899700 1 -- 172.21.15.103:0/3431840035 wait complete. 2023-07-02T09:02:25.385 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.382+0000 7f7a5b899700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.383+0000 7f7a5b899700 1 Processor -- start 2023-07-02T09:02:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.383+0000 7f7a5b899700 1 -- start start 2023-07-02T09:02:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.383+0000 7f7a5b899700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54098a00 0x7f7a541c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.383+0000 7f7a5b899700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54120ac0 con 0x7f7a54098a00 2023-07-02T09:02:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.383+0000 7f7a59635700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54098a00 0x7f7a541c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.383+0000 7f7a59635700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54098a00 0x7f7a541c49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55140/0 (socket says 172.21.15.103:55140) 2023-07-02T09:02:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.383+0000 7f7a59635700 1 -- 172.21.15.103:0/1839917753 learned_addr learned my addr 172.21.15.103:0/1839917753 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:25.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.384+0000 7f7a59635700 1 -- 172.21.15.103:0/1839917753 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5411cc00 con 0x7f7a54098a00 2023-07-02T09:02:25.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.384+0000 7f7a59635700 1 --2- 172.21.15.103:0/1839917753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54098a00 0x7f7a541c49d0 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:25.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.384+0000 7f7a49ffb700 1 -- 172.21.15.103:0/1839917753 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a44005a60 con 0x7f7a54098a00 2023-07-02T09:02:25.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.384+0000 7f7a49ffb700 1 -- 172.21.15.103:0/1839917753 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7a44005bc0 con 0x7f7a54098a00 2023-07-02T09:02:25.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.384+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5411f120 con 0x7f7a54098a00 2023-07-02T09:02:25.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.384+0000 7f7a49ffb700 1 -- 172.21.15.103:0/1839917753 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a44005e60 con 0x7f7a54098a00 2023-07-02T09:02:25.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.384+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5412e030 con 0x7f7a54098a00 2023-07-02T09:02:25.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.385+0000 7f7a49ffb700 1 -- 172.21.15.103:0/1839917753 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7a4400c7b0 con 0x7f7a54098a00 2023-07-02T09:02:25.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.385+0000 7f7a49ffb700 1 --2- 172.21.15.103:0/1839917753 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a380377c0 0x7f7a38039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:25.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.385+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a54061ea0 con 0x7f7a54098a00 2023-07-02T09:02:25.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.385+0000 7f7a49ffb700 1 -- 172.21.15.103:0/1839917753 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7a44046b70 con 0x7f7a54098a00 2023-07-02T09:02:25.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.385+0000 7f7a58e34700 1 --2- 172.21.15.103:0/1839917753 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a380377c0 0x7f7a38039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:25.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.386+0000 7f7a58e34700 1 --2- 172.21.15.103:0/1839917753 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a380377c0 0x7f7a38039cd0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:25.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.390+0000 7f7a49ffb700 1 -- 172.21.15.103:0/1839917753 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7a4400a4a0 con 0x7f7a54098a00 2023-07-02T09:02:25.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~160s 2023-07-02T09:02:25.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.579+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7a54073ca0 con 0x7f7a54098a00 2023-07-02T09:02:25.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.580+0000 7f7a49ffb700 1 -- 172.21.15.103:0/1839917753 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7a4400a4a0 con 0x7f7a54098a00 2023-07-02T09:02:25.581 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:25.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a380377c0 msgr2=0x7f7a38039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:25.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 --2- 172.21.15.103:0/1839917753 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a380377c0 0x7f7a38039cd0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:25.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54098a00 msgr2=0x7f7a541c49d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:25.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 --2- 172.21.15.103:0/1839917753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54098a00 0x7f7a541c49d0 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:25.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 shutdown_connections 2023-07-02T09:02:25.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 --2- 172.21.15.103:0/1839917753 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a380377c0 0x7f7a38039cd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:25.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 --2- 172.21.15.103:0/1839917753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54098a00 0x7f7a541c49d0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:25.584 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 >> 172.21.15.103:0/1839917753 conn(0x7f7a5409d810 msgr2=0x7f7a541306f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:25.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 shutdown_connections 2023-07-02T09:02:25.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:25.582+0000 7f7a5b899700 1 -- 172.21.15.103:0/1839917753 wait complete. 2023-07-02T09:02:25.596 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:26.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.736+0000 7f6f051f8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:26.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.742+0000 7f6f051f8700 1 Processor -- start 2023-07-02T09:02:26.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.742+0000 7f6f051f8700 1 -- start start 2023-07-02T09:02:26.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.742+0000 7f6f051f8700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f00133360 0x7f6f0009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:26.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.742+0000 7f6f051f8700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0009e260 con 0x7f6f00132f50 2023-07-02T09:02:26.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.742+0000 7f6f051f8700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f000a1640 con 0x7f6f00133360 2023-07-02T09:02:26.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.742+0000 7f6efe59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f00133360 0x7f6f0009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:26.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.742+0000 7f6efed9d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6f00132f50 0x7f6f0009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54500/0 (socket says 172.21.15.103:54500) 2023-07-02T09:02:26.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efe59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f00133360 0x7f6f0009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55150/0 (socket says 172.21.15.103:55150) 2023-07-02T09:02:26.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efed9d700 1 -- 172.21.15.103:0/1116302603 learned_addr learned my addr 172.21.15.103:0/1116302603 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:26.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efe59c700 1 -- 172.21.15.103:0/1116302603 >> v1:172.21.15.103:6789/0 conn(0x7f6f00132f50 legacy=0x7f6f0009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:26.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efe59c700 1 -- 172.21.15.103:0/1116302603 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00132a20 con 0x7f6f00133360 2023-07-02T09:02:26.745 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efdd9b700 1 -- 172.21.15.103:0/1116302603 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339776047 0 0) 0x7f6f0009e260 con 0x7f6f00132f50 2023-07-02T09:02:26.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efe59c700 1 --2- 172.21.15.103:0/1116302603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00133360 0x7f6f0009cc90 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b562f95e145ddd4 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:26.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efdd9b700 1 -- 172.21.15.103:0/1116302603 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6ef4009cd0 con 0x7f6f00133360 2023-07-02T09:02:26.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efdd9b700 1 -- 172.21.15.103:0/1116302603 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6ef4009e30 con 0x7f6f00133360 2023-07-02T09:02:26.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.743+0000 7f6efdd9b700 1 -- 172.21.15.103:0/1116302603 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6ef400e360 con 0x7f6f00133360 2023-07-02T09:02:26.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.744+0000 7f6f051f8700 1 -- 172.21.15.103:0/1116302603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00133360 msgr2=0x7f6f0009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:26.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.744+0000 7f6f051f8700 1 --2- 172.21.15.103:0/1116302603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00133360 0x7f6f0009cc90 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:26.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.744+0000 7f6f051f8700 1 -- 172.21.15.103:0/1116302603 shutdown_connections 2023-07-02T09:02:26.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.744+0000 7f6f051f8700 1 --2- 172.21.15.103:0/1116302603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00133360 0x7f6f0009cc90 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:26.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.744+0000 7f6f051f8700 1 -- 172.21.15.103:0/1116302603 >> 172.21.15.103:0/1116302603 conn(0x7f6f000a2080 msgr2=0x7f6f000a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:26.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.744+0000 7f6f051f8700 1 -- 172.21.15.103:0/1116302603 shutdown_connections 2023-07-02T09:02:26.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.744+0000 7f6f051f8700 1 -- 172.21.15.103:0/1116302603 wait complete. 2023-07-02T09:02:26.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.744+0000 7f6f051f8700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:26.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.745+0000 7f6f051f8700 1 Processor -- start 2023-07-02T09:02:26.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.745+0000 7f6f051f8700 1 -- start start 2023-07-02T09:02:26.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.746+0000 7f6f051f8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00132f50 0x7f6f001ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:26.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.746+0000 7f6f051f8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0006d370 con 0x7f6f00132f50 2023-07-02T09:02:26.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.746+0000 7f6efed9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00132f50 0x7f6f001ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:26.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.746+0000 7f6efed9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00132f50 0x7f6f001ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55154/0 (socket says 172.21.15.103:55154) 2023-07-02T09:02:26.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.746+0000 7f6efed9d700 1 -- 172.21.15.103:0/1493168405 learned_addr learned my addr 172.21.15.103:0/1493168405 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:26.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.746+0000 7f6efed9d700 1 -- 172.21.15.103:0/1493168405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0006dde0 con 0x7f6f00132f50 2023-07-02T09:02:26.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.746+0000 7f6efed9d700 1 --2- 172.21.15.103:0/1493168405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00132f50 0x7f6f001ba9b0 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:26.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.746+0000 7f6ee77fe700 1 -- 172.21.15.103:0/1493168405 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6ef0004b80 con 0x7f6f00132f50 2023-07-02T09:02:26.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.747+0000 7f6ee77fe700 1 -- 172.21.15.103:0/1493168405 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6ef0004ce0 con 0x7f6f00132f50 2023-07-02T09:02:26.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.747+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f001ba4f0 con 0x7f6f00132f50 2023-07-02T09:02:26.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.747+0000 7f6ee77fe700 1 -- 172.21.15.103:0/1493168405 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6ef000c4f0 con 0x7f6f00132f50 2023-07-02T09:02:26.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.747+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f00071e40 con 0x7f6f00132f50 2023-07-02T09:02:26.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.748+0000 7f6ee77fe700 1 -- 172.21.15.103:0/1493168405 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6ef000c650 con 0x7f6f00132f50 2023-07-02T09:02:26.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.748+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f00061ea0 con 0x7f6f00132f50 2023-07-02T09:02:26.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.748+0000 7f6ee77fe700 1 --2- 172.21.15.103:0/1493168405 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6eec0377c0 0x7f6eec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:26.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.748+0000 7f6ee77fe700 1 -- 172.21.15.103:0/1493168405 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6ef0046970 con 0x7f6f00132f50 2023-07-02T09:02:26.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.748+0000 7f6efe59c700 1 --2- 172.21.15.103:0/1493168405 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6eec0377c0 0x7f6eec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:26.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.749+0000 7f6efe59c700 1 --2- 172.21.15.103:0/1493168405 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6eec0377c0 0x7f6eec039cd0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:26.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.753+0000 7f6ee77fe700 1 -- 172.21.15.103:0/1493168405 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6ef0009c80 con 0x7f6f00132f50 2023-07-02T09:02:26.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.945+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6f000725b0 con 0x7f6f00132f50 2023-07-02T09:02:26.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.946+0000 7f6ee77fe700 1 -- 172.21.15.103:0/1493168405 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6ef00098a0 con 0x7f6f00132f50 2023-07-02T09:02:26.947 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:26.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6eec0377c0 msgr2=0x7f6eec039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:26.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 --2- 172.21.15.103:0/1493168405 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6eec0377c0 0x7f6eec039cd0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:26.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00132f50 msgr2=0x7f6f001ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:26.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 --2- 172.21.15.103:0/1493168405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00132f50 0x7f6f001ba9b0 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:26.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 shutdown_connections 2023-07-02T09:02:26.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 --2- 172.21.15.103:0/1493168405 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6eec0377c0 0x7f6eec039cd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:26.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 --2- 172.21.15.103:0/1493168405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f00132f50 0x7f6f001ba9b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:26.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 >> 172.21.15.103:0/1493168405 conn(0x7f6f000a2080 msgr2=0x7f6f000a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:26.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 shutdown_connections 2023-07-02T09:02:26.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:26.948+0000 7f6f051f8700 1 -- 172.21.15.103:0/1493168405 wait complete. 2023-07-02T09:02:26.962 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:27.963 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:28.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.093+0000 7f60c6412700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:28.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.098+0000 7f60c6412700 1 Processor -- start 2023-07-02T09:02:28.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.098+0000 7f60c6412700 1 -- start start 2023-07-02T09:02:28.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.098+0000 7f60c6412700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f60c012f250 0x7f60c012f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:28.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60c6412700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c006df60 con 0x7f60c0098a00 2023-07-02T09:02:28.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60c6412700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c009a710 con 0x7f60c012f250 2023-07-02T09:02:28.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60bffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f60c012f250 0x7f60c012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:28.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60bffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f60c012f250 0x7f60c012f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55162/0 (socket says 172.21.15.103:55162) 2023-07-02T09:02:28.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60bffff700 1 -- 172.21.15.103:0/2659608311 learned_addr learned my addr 172.21.15.103:0/2659608311 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:28.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60bf7fe700 1 --1- 172.21.15.103:0/2659608311 >> v1:172.21.15.103:6789/0 conn(0x7f60c0098a00 0x7f60c0098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54516/0 (socket says 172.21.15.103:54516) 2023-07-02T09:02:28.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60bffff700 1 -- 172.21.15.103:0/2659608311 >> v1:172.21.15.103:6789/0 conn(0x7f60c0098a00 legacy=0x7f60c0098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:28.101 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60bffff700 1 -- 172.21.15.103:0/2659608311 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c012ed20 con 0x7f60c012f250 2023-07-02T09:02:28.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60bffff700 1 --2- 172.21.15.103:0/2659608311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c012f250 0x7f60c012f660 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb99a1d8d83da159 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:28.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60beffd700 1 -- 172.21.15.103:0/2659608311 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f60b4009d90 con 0x7f60c012f250 2023-07-02T09:02:28.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60beffd700 1 -- 172.21.15.103:0/2659608311 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f60b400e230 con 0x7f60c012f250 2023-07-02T09:02:28.102 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.099+0000 7f60beffd700 1 -- 172.21.15.103:0/2659608311 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f60b400e4d0 con 0x7f60c012f250 2023-07-02T09:02:28.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.100+0000 7f60c6412700 1 -- 172.21.15.103:0/2659608311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c012f250 msgr2=0x7f60c012f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:28.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.100+0000 7f60c6412700 1 --2- 172.21.15.103:0/2659608311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c012f250 0x7f60c012f660 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:28.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.100+0000 7f60c6412700 1 -- 172.21.15.103:0/2659608311 shutdown_connections 2023-07-02T09:02:28.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.100+0000 7f60c6412700 1 --2- 172.21.15.103:0/2659608311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c012f250 0x7f60c012f660 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:28.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.100+0000 7f60c6412700 1 -- 172.21.15.103:0/2659608311 >> 172.21.15.103:0/2659608311 conn(0x7f60c009d810 msgr2=0x7f60c009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:28.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.100+0000 7f60c6412700 1 -- 172.21.15.103:0/2659608311 shutdown_connections 2023-07-02T09:02:28.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.100+0000 7f60c6412700 1 -- 172.21.15.103:0/2659608311 wait complete. 2023-07-02T09:02:28.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.100+0000 7f60c6412700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:28.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.101+0000 7f60c6412700 1 Processor -- start 2023-07-02T09:02:28.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.101+0000 7f60c6412700 1 -- start start 2023-07-02T09:02:28.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60c6412700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0098a00 0x7f60c01333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:28.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60c6412700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c01347b0 con 0x7f60c0098a00 2023-07-02T09:02:28.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0098a00 0x7f60c01333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:28.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0098a00 0x7f60c01333d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55176/0 (socket says 172.21.15.103:55176) 2023-07-02T09:02:28.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60bffff700 1 -- 172.21.15.103:0/980338919 learned_addr learned my addr 172.21.15.103:0/980338919 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:28.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60bffff700 1 -- 172.21.15.103:0/980338919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c0134480 con 0x7f60c0098a00 2023-07-02T09:02:28.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60bffff700 1 --2- 172.21.15.103:0/980338919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0098a00 0x7f60c01333d0 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:28.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60bcff9700 1 -- 172.21.15.103:0/980338919 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f60b4017040 con 0x7f60c0098a00 2023-07-02T09:02:28.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.102+0000 7f60bcff9700 1 -- 172.21.15.103:0/980338919 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f60b40114b0 con 0x7f60c0098a00 2023-07-02T09:02:28.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.103+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c0117c90 con 0x7f60c0098a00 2023-07-02T09:02:28.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.103+0000 7f60bcff9700 1 -- 172.21.15.103:0/980338919 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f60b4011750 con 0x7f60c0098a00 2023-07-02T09:02:28.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.103+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c0118c40 con 0x7f60c0098a00 2023-07-02T09:02:28.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.103+0000 7f60bcff9700 1 -- 172.21.15.103:0/980338919 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f60b4011900 con 0x7f60c0098a00 2023-07-02T09:02:28.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.103+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c0118000 con 0x7f60c0098a00 2023-07-02T09:02:28.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.104+0000 7f60bcff9700 1 --2- 172.21.15.103:0/980338919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f60ac0377b0 0x7f60ac039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:28.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.104+0000 7f60bcff9700 1 -- 172.21.15.103:0/980338919 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f60b40497a0 con 0x7f60c0098a00 2023-07-02T09:02:28.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.104+0000 7f60bf7fe700 1 --2- 172.21.15.103:0/980338919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f60ac0377b0 0x7f60ac039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:28.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.104+0000 7f60bf7fe700 1 --2- 172.21.15.103:0/980338919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f60ac0377b0 0x7f60ac039cc0 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:28.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.108+0000 7f60bcff9700 1 -- 172.21.15.103:0/980338919 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f60b400c150 con 0x7f60c0098a00 2023-07-02T09:02:28.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.301+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f60c0061ea0 con 0x7f60c0098a00 2023-07-02T09:02:28.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.302+0000 7f60bcff9700 1 -- 172.21.15.103:0/980338919 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f60b4016130 con 0x7f60c0098a00 2023-07-02T09:02:28.304 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:28.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f60ac0377b0 msgr2=0x7f60ac039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:28.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 --2- 172.21.15.103:0/980338919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f60ac0377b0 0x7f60ac039cc0 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:28.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0098a00 msgr2=0x7f60c01333d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:28.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 --2- 172.21.15.103:0/980338919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0098a00 0x7f60c01333d0 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:28.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 shutdown_connections 2023-07-02T09:02:28.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 --2- 172.21.15.103:0/980338919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f60ac0377b0 0x7f60ac039cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:28.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 --2- 172.21.15.103:0/980338919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0098a00 0x7f60c01333d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:28.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 >> 172.21.15.103:0/980338919 conn(0x7f60c009d810 msgr2=0x7f60c012cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:28.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 shutdown_connections 2023-07-02T09:02:28.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:28.304+0000 7f60c6412700 1 -- 172.21.15.103:0/980338919 wait complete. 2023-07-02T09:02:28.318 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:29.319 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:29.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.456+0000 7f4388868700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:29.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.461+0000 7f4388868700 1 Processor -- start 2023-07-02T09:02:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.461+0000 7f4388868700 1 -- start start 2023-07-02T09:02:29.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.462+0000 7f4388868700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4384133360 0x7f438409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:29.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.462+0000 7f4388868700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438409e260 con 0x7f4384132f50 2023-07-02T09:02:29.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.462+0000 7f4388868700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43840a1640 con 0x7f4384133360 2023-07-02T09:02:29.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.462+0000 7f4381d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4384133360 0x7f438409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:29.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.462+0000 7f438259c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4384132f50 0x7f438409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54530/0 (socket says 172.21.15.103:54530) 2023-07-02T09:02:29.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.462+0000 7f4381d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4384133360 0x7f438409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55180/0 (socket says 172.21.15.103:55180) 2023-07-02T09:02:29.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.462+0000 7f438259c700 1 -- 172.21.15.103:0/4061338161 learned_addr learned my addr 172.21.15.103:0/4061338161 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:29.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.463+0000 7f4381d9b700 1 -- 172.21.15.103:0/4061338161 >> v1:172.21.15.103:6789/0 conn(0x7f4384132f50 legacy=0x7f438409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:29.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.463+0000 7f4381d9b700 1 -- 172.21.15.103:0/4061338161 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4384132a20 con 0x7f4384133360 2023-07-02T09:02:29.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.463+0000 7f4381d9b700 1 --2- 172.21.15.103:0/4061338161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384133360 0x7f438409cc90 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737767d47aa77f9c server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:29.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.463+0000 7f438159a700 1 -- 172.21.15.103:0/4061338161 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4374009cd0 con 0x7f4384133360 2023-07-02T09:02:29.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.463+0000 7f438159a700 1 -- 172.21.15.103:0/4061338161 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4374009e30 con 0x7f4384133360 2023-07-02T09:02:29.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.463+0000 7f438159a700 1 -- 172.21.15.103:0/4061338161 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f437400e360 con 0x7f4384133360 2023-07-02T09:02:29.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.464+0000 7f4388868700 1 -- 172.21.15.103:0/4061338161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384133360 msgr2=0x7f438409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:29.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.464+0000 7f4388868700 1 --2- 172.21.15.103:0/4061338161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384133360 0x7f438409cc90 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:29.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.464+0000 7f4388868700 1 -- 172.21.15.103:0/4061338161 shutdown_connections 2023-07-02T09:02:29.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.464+0000 7f4388868700 1 --2- 172.21.15.103:0/4061338161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384133360 0x7f438409cc90 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:29.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.464+0000 7f4388868700 1 -- 172.21.15.103:0/4061338161 >> 172.21.15.103:0/4061338161 conn(0x7f43840a2080 msgr2=0x7f43840a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:29.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.464+0000 7f4388868700 1 -- 172.21.15.103:0/4061338161 shutdown_connections 2023-07-02T09:02:29.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.464+0000 7f4388868700 1 -- 172.21.15.103:0/4061338161 wait complete. 2023-07-02T09:02:29.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.464+0000 7f4388868700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:29.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.465+0000 7f4388868700 1 Processor -- start 2023-07-02T09:02:29.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.465+0000 7f4388868700 1 -- start start 2023-07-02T09:02:29.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f4388868700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384132f50 0x7f438412bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:29.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f4388868700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438406d260 con 0x7f4384132f50 2023-07-02T09:02:29.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f438259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384132f50 0x7f438412bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:29.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f438259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384132f50 0x7f438412bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55190/0 (socket says 172.21.15.103:55190) 2023-07-02T09:02:29.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f438259c700 1 -- 172.21.15.103:0/799186324 learned_addr learned my addr 172.21.15.103:0/799186324 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:29.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f438259c700 1 -- 172.21.15.103:0/799186324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438412bb90 con 0x7f4384132f50 2023-07-02T09:02:29.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f438259c700 1 --2- 172.21.15.103:0/799186324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384132f50 0x7f438412bea0 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:29.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f4372ffd700 1 -- 172.21.15.103:0/799186324 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4378005a60 con 0x7f4384132f50 2023-07-02T09:02:29.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f4372ffd700 1 -- 172.21.15.103:0/799186324 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4378005bc0 con 0x7f4384132f50 2023-07-02T09:02:29.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438406f730 con 0x7f4384132f50 2023-07-02T09:02:29.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.466+0000 7f4372ffd700 1 -- 172.21.15.103:0/799186324 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4378005e60 con 0x7f4384132f50 2023-07-02T09:02:29.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.467+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438406f490 con 0x7f4384132f50 2023-07-02T09:02:29.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.467+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4384061ea0 con 0x7f4384132f50 2023-07-02T09:02:29.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.467+0000 7f4372ffd700 1 -- 172.21.15.103:0/799186324 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f43780070b0 con 0x7f4384132f50 2023-07-02T09:02:29.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.468+0000 7f4372ffd700 1 --2- 172.21.15.103:0/799186324 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43640377c0 0x7f4364039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:29.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.468+0000 7f4372ffd700 1 -- 172.21.15.103:0/799186324 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4378046ab0 con 0x7f4384132f50 2023-07-02T09:02:29.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.468+0000 7f4381d9b700 1 --2- 172.21.15.103:0/799186324 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43640377c0 0x7f4364039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:29.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.468+0000 7f4381d9b700 1 --2- 172.21.15.103:0/799186324 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43640377c0 0x7f4364039cd0 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:29.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.472+0000 7f4372ffd700 1 -- 172.21.15.103:0/799186324 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f437800a4a0 con 0x7f4384132f50 2023-07-02T09:02:29.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.663+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f438406f490 con 0x7f4384132f50 2023-07-02T09:02:29.664 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.663+0000 7f4372ffd700 1 -- 172.21.15.103:0/799186324 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f438406f490 con 0x7f4384132f50 2023-07-02T09:02:29.665 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:29.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43640377c0 msgr2=0x7f4364039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:29.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 --2- 172.21.15.103:0/799186324 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43640377c0 0x7f4364039cd0 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:29.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384132f50 msgr2=0x7f438412bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:29.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 --2- 172.21.15.103:0/799186324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384132f50 0x7f438412bea0 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:29.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 shutdown_connections 2023-07-02T09:02:29.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 --2- 172.21.15.103:0/799186324 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f43640377c0 0x7f4364039cd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:29.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 --2- 172.21.15.103:0/799186324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4384132f50 0x7f438412bea0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:29.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 >> 172.21.15.103:0/799186324 conn(0x7f43840a2080 msgr2=0x7f43841bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:29.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 shutdown_connections 2023-07-02T09:02:29.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:29.665+0000 7f4388868700 1 -- 172.21.15.103:0/799186324 wait complete. 2023-07-02T09:02:29.679 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:30.681 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:30.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.819+0000 7f1673d15700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:30.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.824+0000 7f1673d15700 1 Processor -- start 2023-07-02T09:02:30.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.824+0000 7f1673d15700 1 -- start start 2023-07-02T09:02:30.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.825+0000 7f1673d15700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f166c132f50 0x7f166c133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:30.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.825+0000 7f1673d15700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c09e260 con 0x7f166c09c580 2023-07-02T09:02:30.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.825+0000 7f1673d15700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c0a1640 con 0x7f166c132f50 2023-07-02T09:02:30.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.825+0000 7f1671ab1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f166c132f50 0x7f166c133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:30.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.825+0000 7f16712b0700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f166c09c580 0x7f166c09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54536/0 (socket says 172.21.15.103:54536) 2023-07-02T09:02:30.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.825+0000 7f1671ab1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f166c132f50 0x7f166c133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55206/0 (socket says 172.21.15.103:55206) 2023-07-02T09:02:30.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.825+0000 7f16712b0700 1 -- 172.21.15.103:0/3899571075 learned_addr learned my addr 172.21.15.103:0/3899571075 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:30.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.826+0000 7f1671ab1700 1 -- 172.21.15.103:0/3899571075 >> v1:172.21.15.103:6789/0 conn(0x7f166c09c580 legacy=0x7f166c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:30.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.826+0000 7f1671ab1700 1 -- 172.21.15.103:0/3899571075 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c132a20 con 0x7f166c132f50 2023-07-02T09:02:30.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.826+0000 7f1671ab1700 1 --2- 172.21.15.103:0/3899571075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c132f50 0x7f166c133360 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38cc67e784423cfa server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:30.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.826+0000 7f1670aaf700 1 -- 172.21.15.103:0/3899571075 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f165c009cd0 con 0x7f166c132f50 2023-07-02T09:02:30.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.826+0000 7f1670aaf700 1 -- 172.21.15.103:0/3899571075 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f165c009e30 con 0x7f166c132f50 2023-07-02T09:02:30.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.826+0000 7f1670aaf700 1 -- 172.21.15.103:0/3899571075 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f165c00e360 con 0x7f166c132f50 2023-07-02T09:02:30.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.826+0000 7f1673d15700 1 -- 172.21.15.103:0/3899571075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c132f50 msgr2=0x7f166c133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:30.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.826+0000 7f1673d15700 1 --2- 172.21.15.103:0/3899571075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c132f50 0x7f166c133360 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:30.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.827+0000 7f1673d15700 1 -- 172.21.15.103:0/3899571075 shutdown_connections 2023-07-02T09:02:30.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.827+0000 7f1673d15700 1 --2- 172.21.15.103:0/3899571075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c132f50 0x7f166c133360 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:30.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.827+0000 7f1673d15700 1 -- 172.21.15.103:0/3899571075 >> 172.21.15.103:0/3899571075 conn(0x7f166c0a2080 msgr2=0x7f166c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:30.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.827+0000 7f1673d15700 1 -- 172.21.15.103:0/3899571075 shutdown_connections 2023-07-02T09:02:30.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.827+0000 7f1673d15700 1 -- 172.21.15.103:0/3899571075 wait complete. 2023-07-02T09:02:30.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.827+0000 7f1673d15700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:30.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.828+0000 7f1673d15700 1 Processor -- start 2023-07-02T09:02:30.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.828+0000 7f1673d15700 1 -- start start 2023-07-02T09:02:30.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.828+0000 7f1673d15700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c09c580 0x7f166c12b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:30.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.828+0000 7f1673d15700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c06f9d0 con 0x7f166c09c580 2023-07-02T09:02:30.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.828+0000 7f1671ab1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c09c580 0x7f166c12b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:30.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f1671ab1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c09c580 0x7f166c12b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55220/0 (socket says 172.21.15.103:55220) 2023-07-02T09:02:30.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f1671ab1700 1 -- 172.21.15.103:0/3131293006 learned_addr learned my addr 172.21.15.103:0/3131293006 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:30.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f1671ab1700 1 -- 172.21.15.103:0/3131293006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c133a00 con 0x7f166c09c580 2023-07-02T09:02:30.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f1671ab1700 1 --2- 172.21.15.103:0/3131293006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c09c580 0x7f166c12b7b0 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:30.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f16627fc700 1 -- 172.21.15.103:0/3131293006 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f165c013530 con 0x7f166c09c580 2023-07-02T09:02:30.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f16627fc700 1 -- 172.21.15.103:0/3131293006 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f165c013690 con 0x7f166c09c580 2023-07-02T09:02:30.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c06d160 con 0x7f166c09c580 2023-07-02T09:02:30.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f16627fc700 1 -- 172.21.15.103:0/3131293006 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f165c013930 con 0x7f166c09c580 2023-07-02T09:02:30.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.829+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c06da70 con 0x7f166c09c580 2023-07-02T09:02:30.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.830+0000 7f16627fc700 1 -- 172.21.15.103:0/3131293006 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f165c013ae0 con 0x7f166c09c580 2023-07-02T09:02:30.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.830+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c061ea0 con 0x7f166c09c580 2023-07-02T09:02:30.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.830+0000 7f16627fc700 1 --2- 172.21.15.103:0/3131293006 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f16500377c0 0x7f1650039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:30.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.830+0000 7f16627fc700 1 -- 172.21.15.103:0/3131293006 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f165c018040 con 0x7f166c09c580 2023-07-02T09:02:30.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.830+0000 7f16712b0700 1 --2- 172.21.15.103:0/3131293006 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f16500377c0 0x7f1650039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:30.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.831+0000 7f16712b0700 1 --2- 172.21.15.103:0/3131293006 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f16500377c0 0x7f1650039cd0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:30.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:30.835+0000 7f16627fc700 1 -- 172.21.15.103:0/3131293006 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f165c018e00 con 0x7f166c09c580 2023-07-02T09:02:30.995 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~165s 2023-07-02T09:02:31.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.022+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f166c0725b0 con 0x7f166c09c580 2023-07-02T09:02:31.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.022+0000 7f16627fc700 1 -- 172.21.15.103:0/3131293006 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f165c010e60 con 0x7f166c09c580 2023-07-02T09:02:31.024 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:31.025 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f16500377c0 msgr2=0x7f1650039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:31.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 --2- 172.21.15.103:0/3131293006 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f16500377c0 0x7f1650039cd0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:31.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c09c580 msgr2=0x7f166c12b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:31.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 --2- 172.21.15.103:0/3131293006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c09c580 0x7f166c12b7b0 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:31.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 shutdown_connections 2023-07-02T09:02:31.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 --2- 172.21.15.103:0/3131293006 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f16500377c0 0x7f1650039cd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:31.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 --2- 172.21.15.103:0/3131293006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c09c580 0x7f166c12b7b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:31.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 >> 172.21.15.103:0/3131293006 conn(0x7f166c0a2080 msgr2=0x7f166c1babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:31.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 shutdown_connections 2023-07-02T09:02:31.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:31.024+0000 7f1673d15700 1 -- 172.21.15.103:0/3131293006 wait complete. 2023-07-02T09:02:31.038 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:32.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:32.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.177+0000 7f37d691a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:32.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.182+0000 7f37d691a700 1 Processor -- start 2023-07-02T09:02:32.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.182+0000 7f37d691a700 1 -- start start 2023-07-02T09:02:32.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37d691a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f37d0132f50 0x7f37d0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:32.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37d691a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d009e260 con 0x7f37d009c580 2023-07-02T09:02:32.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37d691a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d00a1640 con 0x7f37d0132f50 2023-07-02T09:02:32.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37cffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f37d0132f50 0x7f37d0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:32.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37cf7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f37d009c580 0x7f37d009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:54548/0 (socket says 172.21.15.103:54548) 2023-07-02T09:02:32.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37cffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f37d0132f50 0x7f37d0133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55228/0 (socket says 172.21.15.103:55228) 2023-07-02T09:02:32.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37cf7fe700 1 -- 172.21.15.103:0/3571970614 learned_addr learned my addr 172.21.15.103:0/3571970614 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:32.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37cffff700 1 -- 172.21.15.103:0/3571970614 >> v1:172.21.15.103:6789/0 conn(0x7f37d009c580 legacy=0x7f37d009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:32.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37cffff700 1 -- 172.21.15.103:0/3571970614 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d0132a20 con 0x7f37d0132f50 2023-07-02T09:02:32.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.183+0000 7f37ceffd700 1 -- 172.21.15.103:0/3571970614 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400135778 0 0) 0x7f37d009e260 con 0x7f37d009c580 2023-07-02T09:02:32.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37cffff700 1 --2- 172.21.15.103:0/3571970614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d0132f50 0x7f37d0133360 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad373f7fd77999de server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:32.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37ceffd700 1 -- 172.21.15.103:0/3571970614 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f37c4009cd0 con 0x7f37d0132f50 2023-07-02T09:02:32.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37ceffd700 1 -- 172.21.15.103:0/3571970614 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f37c4009e30 con 0x7f37d0132f50 2023-07-02T09:02:32.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37ceffd700 1 -- 172.21.15.103:0/3571970614 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f37c400e360 con 0x7f37d0132f50 2023-07-02T09:02:32.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37d691a700 1 -- 172.21.15.103:0/3571970614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d0132f50 msgr2=0x7f37d0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:32.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37d691a700 1 --2- 172.21.15.103:0/3571970614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d0132f50 0x7f37d0133360 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:32.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37d691a700 1 -- 172.21.15.103:0/3571970614 shutdown_connections 2023-07-02T09:02:32.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37d691a700 1 --2- 172.21.15.103:0/3571970614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d0132f50 0x7f37d0133360 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:32.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37d691a700 1 -- 172.21.15.103:0/3571970614 >> 172.21.15.103:0/3571970614 conn(0x7f37d00a2080 msgr2=0x7f37d00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:32.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37d691a700 1 -- 172.21.15.103:0/3571970614 shutdown_connections 2023-07-02T09:02:32.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.184+0000 7f37d691a700 1 -- 172.21.15.103:0/3571970614 wait complete. 2023-07-02T09:02:32.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.185+0000 7f37d691a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:32.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.186+0000 7f37d691a700 1 Processor -- start 2023-07-02T09:02:32.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.186+0000 7f37d691a700 1 -- start start 2023-07-02T09:02:32.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.186+0000 7f37d691a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d009c580 0x7f37d01baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:32.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.186+0000 7f37d691a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d01ba610 con 0x7f37d009c580 2023-07-02T09:02:32.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37cffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d009c580 0x7f37d01baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:32.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37cffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d009c580 0x7f37d01baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55238/0 (socket says 172.21.15.103:55238) 2023-07-02T09:02:32.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37cffff700 1 -- 172.21.15.103:0/4028876812 learned_addr learned my addr 172.21.15.103:0/4028876812 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:32.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37cffff700 1 -- 172.21.15.103:0/4028876812 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d006dfc0 con 0x7f37d009c580 2023-07-02T09:02:32.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37cffff700 1 --2- 172.21.15.103:0/4028876812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d009c580 0x7f37d01baa60 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:32.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37ccff9700 1 -- 172.21.15.103:0/4028876812 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f37c4013530 con 0x7f37d009c580 2023-07-02T09:02:32.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37ccff9700 1 -- 172.21.15.103:0/4028876812 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f37c4013690 con 0x7f37d009c580 2023-07-02T09:02:32.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d01ba270 con 0x7f37d009c580 2023-07-02T09:02:32.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37ccff9700 1 -- 172.21.15.103:0/4028876812 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f37c4013930 con 0x7f37d009c580 2023-07-02T09:02:32.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.187+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d006dba0 con 0x7f37d009c580 2023-07-02T09:02:32.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.188+0000 7f37ccff9700 1 -- 172.21.15.103:0/4028876812 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f37c4013ae0 con 0x7f37d009c580 2023-07-02T09:02:32.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.188+0000 7f37ccff9700 1 --2- 172.21.15.103:0/4028876812 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f37bc0377b0 0x7f37bc039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:32.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.188+0000 7f37ccff9700 1 -- 172.21.15.103:0/4028876812 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f37c4018040 con 0x7f37d009c580 2023-07-02T09:02:32.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.188+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d01ba460 con 0x7f37d009c580 2023-07-02T09:02:32.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.189+0000 7f37cf7fe700 1 --2- 172.21.15.103:0/4028876812 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f37bc0377b0 0x7f37bc039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:32.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.189+0000 7f37cf7fe700 1 --2- 172.21.15.103:0/4028876812 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f37bc0377b0 0x7f37bc039cc0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:32.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.193+0000 7f37ccff9700 1 -- 172.21.15.103:0/4028876812 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f37c4018e00 con 0x7f37d009c580 2023-07-02T09:02:32.385 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.384+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f37d0061ea0 con 0x7f37d009c580 2023-07-02T09:02:32.386 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.384+0000 7f37ccff9700 1 -- 172.21.15.103:0/4028876812 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f37c4010e60 con 0x7f37d009c580 2023-07-02T09:02:32.386 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:32.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f37bc0377b0 msgr2=0x7f37bc039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:32.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 --2- 172.21.15.103:0/4028876812 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f37bc0377b0 0x7f37bc039cc0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:32.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d009c580 msgr2=0x7f37d01baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:32.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 --2- 172.21.15.103:0/4028876812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d009c580 0x7f37d01baa60 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:32.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 shutdown_connections 2023-07-02T09:02:32.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 --2- 172.21.15.103:0/4028876812 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f37bc0377b0 0x7f37bc039cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:32.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 --2- 172.21.15.103:0/4028876812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37d009c580 0x7f37d01baa60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:32.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 >> 172.21.15.103:0/4028876812 conn(0x7f37d00a2080 msgr2=0x7f37d0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:32.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 shutdown_connections 2023-07-02T09:02:32.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:32.387+0000 7f37d691a700 1 -- 172.21.15.103:0/4028876812 wait complete. 2023-07-02T09:02:32.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:33.405 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:33.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.543+0000 7fdaf47d6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:33.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.548+0000 7fdaf47d6700 1 Processor -- start 2023-07-02T09:02:33.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.548+0000 7fdaf47d6700 1 -- start start 2023-07-02T09:02:33.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf47d6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdaec132030 0x7fdaec134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:33.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf47d6700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec06df60 con 0x7fdaec12f250 2023-07-02T09:02:33.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf47d6700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaec09a710 con 0x7fdaec132030 2023-07-02T09:02:33.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1d71700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdaec132030 0x7fdaec134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:33.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf2572700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdaec12f250 0x7fdaec12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58724/0 (socket says 172.21.15.103:58724) 2023-07-02T09:02:33.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1d71700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdaec132030 0x7fdaec134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48314/0 (socket says 172.21.15.103:48314) 2023-07-02T09:02:33.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf2572700 1 -- 172.21.15.103:0/2150445969 learned_addr learned my addr 172.21.15.103:0/2150445969 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:33.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1d71700 1 -- 172.21.15.103:0/2150445969 >> v1:172.21.15.103:6789/0 conn(0x7fdaec12f250 legacy=0x7fdaec12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:33.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1d71700 1 -- 172.21.15.103:0/2150445969 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec12ed20 con 0x7fdaec132030 2023-07-02T09:02:33.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1570700 1 -- 172.21.15.103:0/2150445969 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094814878 0 0) 0x7fdaec06df60 con 0x7fdaec12f250 2023-07-02T09:02:33.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1d71700 1 --2- 172.21.15.103:0/2150445969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec132030 0x7fdaec134450 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5d3910e729a471 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:33.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1570700 1 -- 172.21.15.103:0/2150445969 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdadc009cd0 con 0x7fdaec132030 2023-07-02T09:02:33.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1570700 1 -- 172.21.15.103:0/2150445969 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdadc009e30 con 0x7fdaec132030 2023-07-02T09:02:33.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.549+0000 7fdaf1570700 1 -- 172.21.15.103:0/2150445969 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdadc00e360 con 0x7fdaec132030 2023-07-02T09:02:33.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.550+0000 7fdaf47d6700 1 -- 172.21.15.103:0/2150445969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec132030 msgr2=0x7fdaec134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:33.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.550+0000 7fdaf47d6700 1 --2- 172.21.15.103:0/2150445969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec132030 0x7fdaec134450 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:33.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.550+0000 7fdaf47d6700 1 -- 172.21.15.103:0/2150445969 shutdown_connections 2023-07-02T09:02:33.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.550+0000 7fdaf47d6700 1 --2- 172.21.15.103:0/2150445969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec132030 0x7fdaec134450 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:33.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.550+0000 7fdaf47d6700 1 -- 172.21.15.103:0/2150445969 >> 172.21.15.103:0/2150445969 conn(0x7fdaec09d810 msgr2=0x7fdaec09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:33.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.550+0000 7fdaf47d6700 1 -- 172.21.15.103:0/2150445969 shutdown_connections 2023-07-02T09:02:33.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.550+0000 7fdaf47d6700 1 -- 172.21.15.103:0/2150445969 wait complete. 2023-07-02T09:02:33.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.550+0000 7fdaf47d6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:33.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.551+0000 7fdaf47d6700 1 Processor -- start 2023-07-02T09:02:33.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.551+0000 7fdaf47d6700 1 -- start start 2023-07-02T09:02:33.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.552+0000 7fdaf47d6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec12f250 0x7fdaec1c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:33.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.552+0000 7fdaf47d6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec11a430 con 0x7fdaec12f250 2023-07-02T09:02:33.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.552+0000 7fdaf2572700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec12f250 0x7fdaec1c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:33.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.552+0000 7fdaf2572700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec12f250 0x7fdaec1c5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48322/0 (socket says 172.21.15.103:48322) 2023-07-02T09:02:33.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.552+0000 7fdaf2572700 1 -- 172.21.15.103:0/3630924402 learned_addr learned my addr 172.21.15.103:0/3630924402 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:33.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.552+0000 7fdaf2572700 1 -- 172.21.15.103:0/3630924402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec1292e0 con 0x7fdaec12f250 2023-07-02T09:02:33.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.552+0000 7fdaf2572700 1 --2- 172.21.15.103:0/3630924402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec12f250 0x7fdaec1c5230 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:33.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.552+0000 7fdae2ffd700 1 -- 172.21.15.103:0/3630924402 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdae8004b80 con 0x7fdaec12f250 2023-07-02T09:02:33.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.553+0000 7fdae2ffd700 1 -- 172.21.15.103:0/3630924402 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdae8004ce0 con 0x7fdaec12f250 2023-07-02T09:02:33.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.553+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec119e50 con 0x7fdaec12f250 2023-07-02T09:02:33.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.553+0000 7fdae2ffd700 1 -- 172.21.15.103:0/3630924402 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdae800c4f0 con 0x7fdaec12f250 2023-07-02T09:02:33.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.553+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec119670 con 0x7fdaec12f250 2023-07-02T09:02:33.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.553+0000 7fdae2ffd700 1 -- 172.21.15.103:0/3630924402 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fdae800c650 con 0x7fdaec12f250 2023-07-02T09:02:33.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.553+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec061ea0 con 0x7fdaec12f250 2023-07-02T09:02:33.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.554+0000 7fdae2ffd700 1 --2- 172.21.15.103:0/3630924402 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdad00377c0 0x7fdad0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:33.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.554+0000 7fdae2ffd700 1 -- 172.21.15.103:0/3630924402 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fdae8046900 con 0x7fdaec12f250 2023-07-02T09:02:33.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.554+0000 7fdaf1d71700 1 --2- 172.21.15.103:0/3630924402 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdad00377c0 0x7fdad0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:33.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.554+0000 7fdaf1d71700 1 --2- 172.21.15.103:0/3630924402 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdad00377c0 0x7fdad0039cd0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:33.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.559+0000 7fdae2ffd700 1 -- 172.21.15.103:0/3630924402 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fdae8009c80 con 0x7fdaec12f250 2023-07-02T09:02:33.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.750+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdaec073ca0 con 0x7fdaec12f250 2023-07-02T09:02:33.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.750+0000 7fdae2ffd700 1 -- 172.21.15.103:0/3630924402 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fdae80098a0 con 0x7fdaec12f250 2023-07-02T09:02:33.752 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:33.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdad00377c0 msgr2=0x7fdad0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:33.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 --2- 172.21.15.103:0/3630924402 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdad00377c0 0x7fdad0039cd0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:33.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec12f250 msgr2=0x7fdaec1c5230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:33.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 --2- 172.21.15.103:0/3630924402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec12f250 0x7fdaec1c5230 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:33.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 shutdown_connections 2023-07-02T09:02:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 --2- 172.21.15.103:0/3630924402 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdad00377c0 0x7fdad0039cd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 --2- 172.21.15.103:0/3630924402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec12f250 0x7fdaec1c5230 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 >> 172.21.15.103:0/3630924402 conn(0x7fdaec09d810 msgr2=0x7fdaec12d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 shutdown_connections 2023-07-02T09:02:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:33.752+0000 7fdaf47d6700 1 -- 172.21.15.103:0/3630924402 wait complete. 2023-07-02T09:02:33.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:34.767 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:34.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.905+0000 7f10c674a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:34.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.910+0000 7f10c674a700 1 Processor -- start 2023-07-02T09:02:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.911+0000 7f10c674a700 1 -- start start 2023-07-02T09:02:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.911+0000 7f10c674a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f10c0133360 0x7f10c009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.911+0000 7f10c674a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c009e260 con 0x7f10c0132f50 2023-07-02T09:02:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.911+0000 7f10c674a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c00a1640 con 0x7f10c0133360 2023-07-02T09:02:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.911+0000 7f10bf7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f10c0133360 0x7f10c009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.911+0000 7f10bffff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f10c0132f50 0x7f10c009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58736/0 (socket says 172.21.15.103:58736) 2023-07-02T09:02:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.911+0000 7f10bffff700 1 -- 172.21.15.103:0/2289656392 learned_addr learned my addr 172.21.15.103:0/2289656392 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:34.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.912+0000 7f10bf7fe700 1 -- 172.21.15.103:0/2289656392 >> v1:172.21.15.103:6789/0 conn(0x7f10c0132f50 legacy=0x7f10c009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.912+0000 7f10bf7fe700 1 -- 172.21.15.103:0/2289656392 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c0132a20 con 0x7f10c0133360 2023-07-02T09:02:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.912+0000 7f10beffd700 1 -- 172.21.15.103:0/2289656392 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867019845 0 0) 0x7f10c009e260 con 0x7f10c0132f50 2023-07-02T09:02:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.912+0000 7f10bf7fe700 1 --2- 172.21.15.103:0/2289656392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0133360 0x7f10c009cc90 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27aa7ab3b44b2e0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.912+0000 7f10beffd700 1 -- 172.21.15.103:0/2289656392 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f10b0009cd0 con 0x7f10c0133360 2023-07-02T09:02:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.912+0000 7f10beffd700 1 -- 172.21.15.103:0/2289656392 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f10b0009e30 con 0x7f10c0133360 2023-07-02T09:02:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.912+0000 7f10beffd700 1 -- 172.21.15.103:0/2289656392 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f10b000e360 con 0x7f10c0133360 2023-07-02T09:02:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.913+0000 7f10c674a700 1 -- 172.21.15.103:0/2289656392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0133360 msgr2=0x7f10c009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.913+0000 7f10c674a700 1 --2- 172.21.15.103:0/2289656392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0133360 0x7f10c009cc90 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.913+0000 7f10c674a700 1 -- 172.21.15.103:0/2289656392 shutdown_connections 2023-07-02T09:02:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.913+0000 7f10c674a700 1 --2- 172.21.15.103:0/2289656392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0133360 0x7f10c009cc90 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.913+0000 7f10c674a700 1 -- 172.21.15.103:0/2289656392 >> 172.21.15.103:0/2289656392 conn(0x7f10c00a2080 msgr2=0x7f10c00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.913+0000 7f10c674a700 1 -- 172.21.15.103:0/2289656392 shutdown_connections 2023-07-02T09:02:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.913+0000 7f10c674a700 1 -- 172.21.15.103:0/2289656392 wait complete. 2023-07-02T09:02:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.913+0000 7f10c674a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:34.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.914+0000 7f10c674a700 1 Processor -- start 2023-07-02T09:02:34.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.914+0000 7f10c674a700 1 -- start start 2023-07-02T09:02:34.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10c674a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0132f50 0x7f10c012c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:34.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10c674a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c006d2f0 con 0x7f10c0132f50 2023-07-02T09:02:34.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0132f50 0x7f10c012c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:34.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0132f50 0x7f10c012c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48340/0 (socket says 172.21.15.103:48340) 2023-07-02T09:02:34.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10bffff700 1 -- 172.21.15.103:0/1130137612 learned_addr learned my addr 172.21.15.103:0/1130137612 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:34.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10bffff700 1 -- 172.21.15.103:0/1130137612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c006dd60 con 0x7f10c0132f50 2023-07-02T09:02:34.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10bffff700 1 --2- 172.21.15.103:0/1130137612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0132f50 0x7f10c012c1a0 crc :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:34.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10bcff9700 1 -- 172.21.15.103:0/1130137612 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f10b4004b80 con 0x7f10c0132f50 2023-07-02T09:02:34.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10bcff9700 1 -- 172.21.15.103:0/1130137612 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f10b4004ce0 con 0x7f10c0132f50 2023-07-02T09:02:34.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c012bc50 con 0x7f10c0132f50 2023-07-02T09:02:34.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10bcff9700 1 -- 172.21.15.103:0/1130137612 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f10b400c4f0 con 0x7f10c0132f50 2023-07-02T09:02:34.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.915+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c006f850 con 0x7f10c0132f50 2023-07-02T09:02:34.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.916+0000 7f10bcff9700 1 -- 172.21.15.103:0/1130137612 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f10b400c650 con 0x7f10c0132f50 2023-07-02T09:02:34.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.916+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c006dbb0 con 0x7f10c0132f50 2023-07-02T09:02:34.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.917+0000 7f10bcff9700 1 --2- 172.21.15.103:0/1130137612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f10ac0377c0 0x7f10ac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:34.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.917+0000 7f10bcff9700 1 -- 172.21.15.103:0/1130137612 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f10b4046970 con 0x7f10c0132f50 2023-07-02T09:02:34.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.917+0000 7f10bf7fe700 1 --2- 172.21.15.103:0/1130137612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f10ac0377c0 0x7f10ac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:34.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.917+0000 7f10bf7fe700 1 --2- 172.21.15.103:0/1130137612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f10ac0377c0 0x7f10ac039cd0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:34.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:34.921+0000 7f10bcff9700 1 -- 172.21.15.103:0/1130137612 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f10b4009c80 con 0x7f10c0132f50 2023-07-02T09:02:35.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.113+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f10c0061ea0 con 0x7f10c0132f50 2023-07-02T09:02:35.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.113+0000 7f10bcff9700 1 -- 172.21.15.103:0/1130137612 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f10b40098a0 con 0x7f10c0132f50 2023-07-02T09:02:35.115 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:35.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.115+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f10ac0377c0 msgr2=0x7f10ac039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:35.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.115+0000 7f10c674a700 1 --2- 172.21.15.103:0/1130137612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f10ac0377c0 0x7f10ac039cd0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:35.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.115+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0132f50 msgr2=0x7f10c012c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:35.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.115+0000 7f10c674a700 1 --2- 172.21.15.103:0/1130137612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0132f50 0x7f10c012c1a0 crc :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:35.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.116+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 shutdown_connections 2023-07-02T09:02:35.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.116+0000 7f10c674a700 1 --2- 172.21.15.103:0/1130137612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f10ac0377c0 0x7f10ac039cd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:35.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.116+0000 7f10c674a700 1 --2- 172.21.15.103:0/1130137612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10c0132f50 0x7f10c012c1a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:35.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.116+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 >> 172.21.15.103:0/1130137612 conn(0x7f10c00a2080 msgr2=0x7f10c0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:35.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.116+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 shutdown_connections 2023-07-02T09:02:35.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:35.116+0000 7f10c674a700 1 -- 172.21.15.103:0/1130137612 wait complete. 2023-07-02T09:02:35.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:36.131 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:36.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.267+0000 7f7827965700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:36.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.271+0000 7f7827965700 1 Processor -- start 2023-07-02T09:02:36.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.272+0000 7f7827965700 1 -- start start 2023-07-02T09:02:36.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.272+0000 7f7827965700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f782012f250 0x7f782012f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:36.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.272+0000 7f7827965700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782006df60 con 0x7f7820098a00 2023-07-02T09:02:36.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.272+0000 7f7827965700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782009a710 con 0x7f782012f250 2023-07-02T09:02:36.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7825701700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f782012f250 0x7f782012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:36.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7824f00700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7820098a00 0x7f7820098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58748/0 (socket says 172.21.15.103:58748) 2023-07-02T09:02:36.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7825701700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f782012f250 0x7f782012f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48342/0 (socket says 172.21.15.103:48342) 2023-07-02T09:02:36.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7824f00700 1 -- 172.21.15.103:0/2655286199 learned_addr learned my addr 172.21.15.103:0/2655286199 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:36.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7825701700 1 -- 172.21.15.103:0/2655286199 >> v1:172.21.15.103:6789/0 conn(0x7f7820098a00 legacy=0x7f7820098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:36.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7817fff700 1 -- 172.21.15.103:0/2655286199 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999926622 0 0) 0x7f782006df60 con 0x7f7820098a00 2023-07-02T09:02:36.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7825701700 1 -- 172.21.15.103:0/2655286199 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782012ed20 con 0x7f782012f250 2023-07-02T09:02:36.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7825701700 1 --2- 172.21.15.103:0/2655286199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f782012f250 0x7f782012f660 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e2eaf927d6b4a5 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:36.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7817fff700 1 -- 172.21.15.103:0/2655286199 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f781000ac60 con 0x7f782012f250 2023-07-02T09:02:36.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7817fff700 1 -- 172.21.15.103:0/2655286199 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f781000adc0 con 0x7f782012f250 2023-07-02T09:02:36.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.273+0000 7f7817fff700 1 -- 172.21.15.103:0/2655286199 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f781000f470 con 0x7f782012f250 2023-07-02T09:02:36.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.274+0000 7f7827965700 1 -- 172.21.15.103:0/2655286199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f782012f250 msgr2=0x7f782012f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:36.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.274+0000 7f7827965700 1 --2- 172.21.15.103:0/2655286199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f782012f250 0x7f782012f660 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:36.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.274+0000 7f7827965700 1 -- 172.21.15.103:0/2655286199 shutdown_connections 2023-07-02T09:02:36.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.274+0000 7f7827965700 1 --2- 172.21.15.103:0/2655286199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f782012f250 0x7f782012f660 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:36.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.274+0000 7f7827965700 1 -- 172.21.15.103:0/2655286199 >> 172.21.15.103:0/2655286199 conn(0x7f782009d810 msgr2=0x7f782009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:36.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.274+0000 7f7827965700 1 -- 172.21.15.103:0/2655286199 shutdown_connections 2023-07-02T09:02:36.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.274+0000 7f7827965700 1 -- 172.21.15.103:0/2655286199 wait complete. 2023-07-02T09:02:36.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.274+0000 7f7827965700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:36.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.275+0000 7f7827965700 1 Processor -- start 2023-07-02T09:02:36.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.275+0000 7f7827965700 1 -- start start 2023-07-02T09:02:36.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7827965700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7820098a00 0x7f7820119570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:36.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7827965700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782011b970 con 0x7f7820098a00 2023-07-02T09:02:36.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7825701700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7820098a00 0x7f7820119570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:36.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7825701700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7820098a00 0x7f7820119570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48358/0 (socket says 172.21.15.103:48358) 2023-07-02T09:02:36.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7825701700 1 -- 172.21.15.103:0/1248418761 learned_addr learned my addr 172.21.15.103:0/1248418761 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:36.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7825701700 1 -- 172.21.15.103:0/1248418761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782011c8a0 con 0x7f7820098a00 2023-07-02T09:02:36.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7825701700 1 --2- 172.21.15.103:0/1248418761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7820098a00 0x7f7820119570 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:36.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7815ffb700 1 -- 172.21.15.103:0/1248418761 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7810011420 con 0x7f7820098a00 2023-07-02T09:02:36.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7815ffb700 1 -- 172.21.15.103:0/1248418761 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7810011580 con 0x7f7820098a00 2023-07-02T09:02:36.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7815ffb700 1 -- 172.21.15.103:0/1248418761 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7810011820 con 0x7f7820098a00 2023-07-02T09:02:36.281 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.276+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782011aa40 con 0x7f7820098a00 2023-07-02T09:02:36.281 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.277+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782011d3d0 con 0x7f7820098a00 2023-07-02T09:02:36.281 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.277+0000 7f7815ffb700 1 -- 172.21.15.103:0/1248418761 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f78100119d0 con 0x7f7820098a00 2023-07-02T09:02:36.282 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.277+0000 7f7815ffb700 1 --2- 172.21.15.103:0/1248418761 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78040377c0 0x7f7804039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:36.282 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.277+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7820061ea0 con 0x7f7820098a00 2023-07-02T09:02:36.282 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.278+0000 7f7815ffb700 1 -- 172.21.15.103:0/1248418761 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7810047660 con 0x7f7820098a00 2023-07-02T09:02:36.282 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.278+0000 7f7824f00700 1 --2- 172.21.15.103:0/1248418761 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78040377c0 0x7f7804039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:36.282 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.278+0000 7f7824f00700 1 --2- 172.21.15.103:0/1248418761 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78040377c0 0x7f7804039cd0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:36.283 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.282+0000 7f7815ffb700 1 -- 172.21.15.103:0/1248418761 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f78100103b0 con 0x7f7820098a00 2023-07-02T09:02:36.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.473+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f782011aa40 con 0x7f7820098a00 2023-07-02T09:02:36.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.474+0000 7f7815ffb700 1 -- 172.21.15.103:0/1248418761 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f782011aa40 con 0x7f7820098a00 2023-07-02T09:02:36.475 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:36.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78040377c0 msgr2=0x7f7804039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:36.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 --2- 172.21.15.103:0/1248418761 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78040377c0 0x7f7804039cd0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:36.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7820098a00 msgr2=0x7f7820119570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:36.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 --2- 172.21.15.103:0/1248418761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7820098a00 0x7f7820119570 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:36.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 shutdown_connections 2023-07-02T09:02:36.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 --2- 172.21.15.103:0/1248418761 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f78040377c0 0x7f7804039cd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:36.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 --2- 172.21.15.103:0/1248418761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7820098a00 0x7f7820119570 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:36.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 >> 172.21.15.103:0/1248418761 conn(0x7f782009d810 msgr2=0x7f78200a0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:36.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 shutdown_connections 2023-07-02T09:02:36.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:36.476+0000 7f7827965700 1 -- 172.21.15.103:0/1248418761 wait complete. 2023-07-02T09:02:36.490 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:36.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~171s 2023-07-02T09:02:37.491 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:37.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.629+0000 7fa795ded700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:37.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.634+0000 7fa795ded700 1 Processor -- start 2023-07-02T09:02:37.636 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.634+0000 7fa795ded700 1 -- start start 2023-07-02T09:02:37.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa795ded700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa790132030 0x7fa790134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:37.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa795ded700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa79006df60 con 0x7fa79012f250 2023-07-02T09:02:37.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa795ded700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79009a710 con 0x7fa790132030 2023-07-02T09:02:37.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa78effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa790132030 0x7fa790134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:37.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa78f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa79012f250 0x7fa79012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58760/0 (socket says 172.21.15.103:58760) 2023-07-02T09:02:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa78effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa790132030 0x7fa790134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48372/0 (socket says 172.21.15.103:48372) 2023-07-02T09:02:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa78f7fe700 1 -- 172.21.15.103:0/251758391 learned_addr learned my addr 172.21.15.103:0/251758391 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa78effd700 1 -- 172.21.15.103:0/251758391 >> v1:172.21.15.103:6789/0 conn(0x7fa79012f250 legacy=0x7fa79012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.635+0000 7fa78effd700 1 -- 172.21.15.103:0/251758391 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79012ed20 con 0x7fa790132030 2023-07-02T09:02:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa78effd700 1 --2- 172.21.15.103:0/251758391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa790132030 0x7fa790134450 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb04eeb595c0fe0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa78e7fc700 1 -- 172.21.15.103:0/251758391 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa780009cd0 con 0x7fa790132030 2023-07-02T09:02:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa78e7fc700 1 -- 172.21.15.103:0/251758391 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa780009e30 con 0x7fa790132030 2023-07-02T09:02:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa78e7fc700 1 -- 172.21.15.103:0/251758391 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa78000e360 con 0x7fa790132030 2023-07-02T09:02:37.639 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa795ded700 1 -- 172.21.15.103:0/251758391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa790132030 msgr2=0x7fa790134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa795ded700 1 --2- 172.21.15.103:0/251758391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa790132030 0x7fa790134450 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa795ded700 1 -- 172.21.15.103:0/251758391 shutdown_connections 2023-07-02T09:02:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa795ded700 1 --2- 172.21.15.103:0/251758391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa790132030 0x7fa790134450 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa795ded700 1 -- 172.21.15.103:0/251758391 >> 172.21.15.103:0/251758391 conn(0x7fa79009d810 msgr2=0x7fa79009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.636+0000 7fa795ded700 1 -- 172.21.15.103:0/251758391 shutdown_connections 2023-07-02T09:02:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.637+0000 7fa795ded700 1 -- 172.21.15.103:0/251758391 wait complete. 2023-07-02T09:02:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.637+0000 7fa795ded700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:37.641 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.638+0000 7fa795ded700 1 Processor -- start 2023-07-02T09:02:37.641 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.638+0000 7fa795ded700 1 -- start start 2023-07-02T09:02:37.642 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.639+0000 7fa795ded700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa79012f250 0x7fa790070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:37.642 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.639+0000 7fa795ded700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa790071b50 con 0x7fa79012f250 2023-07-02T09:02:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.639+0000 7fa78f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa79012f250 0x7fa790070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.639+0000 7fa78f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa79012f250 0x7fa790070710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48384/0 (socket says 172.21.15.103:48384) 2023-07-02T09:02:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.639+0000 7fa78f7fe700 1 -- 172.21.15.103:0/840209063 learned_addr learned my addr 172.21.15.103:0/840209063 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.639+0000 7fa78f7fe700 1 -- 172.21.15.103:0/840209063 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790119220 con 0x7fa79012f250 2023-07-02T09:02:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.640+0000 7fa78f7fe700 1 --2- 172.21.15.103:0/840209063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa79012f250 0x7fa790070710 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.640+0000 7fa787fff700 1 -- 172.21.15.103:0/840209063 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa778005a60 con 0x7fa79012f250 2023-07-02T09:02:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.640+0000 7fa787fff700 1 -- 172.21.15.103:0/840209063 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa778005bc0 con 0x7fa79012f250 2023-07-02T09:02:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.640+0000 7fa787fff700 1 -- 172.21.15.103:0/840209063 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa778005e60 con 0x7fa79012f250 2023-07-02T09:02:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.640+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79009caa0 con 0x7fa79012f250 2023-07-02T09:02:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.640+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa790118ce0 con 0x7fa79012f250 2023-07-02T09:02:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.641+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa790131990 con 0x7fa79012f250 2023-07-02T09:02:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.641+0000 7fa787fff700 1 -- 172.21.15.103:0/840209063 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa7780070b0 con 0x7fa79012f250 2023-07-02T09:02:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.641+0000 7fa787fff700 1 --2- 172.21.15.103:0/840209063 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7700377c0 0x7fa770039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.641+0000 7fa787fff700 1 -- 172.21.15.103:0/840209063 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa778046b20 con 0x7fa79012f250 2023-07-02T09:02:37.645 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.641+0000 7fa78effd700 1 --2- 172.21.15.103:0/840209063 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7700377c0 0x7fa770039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:37.645 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.642+0000 7fa78effd700 1 --2- 172.21.15.103:0/840209063 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7700377c0 0x7fa770039cd0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:37.646 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.646+0000 7fa787fff700 1 -- 172.21.15.103:0/840209063 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa77800fdf0 con 0x7fa79012f250 2023-07-02T09:02:37.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.839+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa790061ea0 con 0x7fa79012f250 2023-07-02T09:02:37.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.839+0000 7fa787fff700 1 -- 172.21.15.103:0/840209063 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa77800ddd0 con 0x7fa79012f250 2023-07-02T09:02:37.841 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:37.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7700377c0 msgr2=0x7fa770039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:37.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 --2- 172.21.15.103:0/840209063 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7700377c0 0x7fa770039cd0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:37.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa79012f250 msgr2=0x7fa790070710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:37.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 --2- 172.21.15.103:0/840209063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa79012f250 0x7fa790070710 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 shutdown_connections 2023-07-02T09:02:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 --2- 172.21.15.103:0/840209063 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa7700377c0 0x7fa770039cd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 --2- 172.21.15.103:0/840209063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa79012f250 0x7fa790070710 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 >> 172.21.15.103:0/840209063 conn(0x7fa79009d810 msgr2=0x7fa79012d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 shutdown_connections 2023-07-02T09:02:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:37.842+0000 7fa795ded700 1 -- 172.21.15.103:0/840209063 wait complete. 2023-07-02T09:02:37.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:38.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:38.999 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:38.995+0000 7f872b0b3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:39.000 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f872b0b3700 1 Processor -- start 2023-07-02T09:02:39.000 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f872b0b3700 1 -- start start 2023-07-02T09:02:39.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f872b0b3700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f872409a220 0x7f87241348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:39.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f872b0b3700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872409cea0 con 0x7f872406d6e0 2023-07-02T09:02:39.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f872b0b3700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872406d490 con 0x7f872409a220 2023-07-02T09:02:39.001 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f8728e4f700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f872406d6e0 0x7f8724099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58772/0 (socket says 172.21.15.103:58772) 2023-07-02T09:02:39.002 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f8728e4f700 1 -- 172.21.15.103:0/74498888 learned_addr learned my addr 172.21.15.103:0/74498888 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:39.002 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f8723fff700 1 --2- 172.21.15.103:0/74498888 >> v2:172.21.15.103:3300/0 conn(0x7f872409a220 0x7f87241348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:39.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f8723fff700 1 -- 172.21.15.103:0/74498888 >> v1:172.21.15.103:6789/0 conn(0x7f872406d6e0 legacy=0x7f8724099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:39.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f8723fff700 1 -- 172.21.15.103:0/74498888 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872406dc80 con 0x7f872409a220 2023-07-02T09:02:39.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f87237fe700 1 -- 172.21.15.103:0/74498888 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906056802 0 0) 0x7f872409cea0 con 0x7f872406d6e0 2023-07-02T09:02:39.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f8723fff700 1 --2- 172.21.15.103:0/74498888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872409a220 0x7f87241348e0 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c187eb7a7ef34d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:39.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f87237fe700 1 -- 172.21.15.103:0/74498888 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8710009cd0 con 0x7f872409a220 2023-07-02T09:02:39.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.000+0000 7f87237fe700 1 -- 172.21.15.103:0/74498888 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8710009e30 con 0x7f872409a220 2023-07-02T09:02:39.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f87237fe700 1 -- 172.21.15.103:0/74498888 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f871000e360 con 0x7f872409a220 2023-07-02T09:02:39.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f872b0b3700 1 -- 172.21.15.103:0/74498888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872409a220 msgr2=0x7f87241348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:39.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f872b0b3700 1 --2- 172.21.15.103:0/74498888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872409a220 0x7f87241348e0 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:39.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f872b0b3700 1 -- 172.21.15.103:0/74498888 shutdown_connections 2023-07-02T09:02:39.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f872b0b3700 1 --2- 172.21.15.103:0/74498888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872409a220 0x7f87241348e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:39.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f872b0b3700 1 -- 172.21.15.103:0/74498888 >> 172.21.15.103:0/74498888 conn(0x7f872409d810 msgr2=0x7f872409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:39.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f872b0b3700 1 -- 172.21.15.103:0/74498888 shutdown_connections 2023-07-02T09:02:39.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f872b0b3700 1 -- 172.21.15.103:0/74498888 wait complete. 2023-07-02T09:02:39.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.001+0000 7f872b0b3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:39.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.002+0000 7f872b0b3700 1 Processor -- start 2023-07-02T09:02:39.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.002+0000 7f872b0b3700 1 -- start start 2023-07-02T09:02:39.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.003+0000 7f872b0b3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872406d6e0 0x7f87241201a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:39.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.003+0000 7f872b0b3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8724122e00 con 0x7f872406d6e0 2023-07-02T09:02:39.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.003+0000 7f8728e4f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872406d6e0 0x7f87241201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:39.007 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.003+0000 7f8728e4f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872406d6e0 0x7f87241201a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48400/0 (socket says 172.21.15.103:48400) 2023-07-02T09:02:39.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.003+0000 7f8728e4f700 1 -- 172.21.15.103:0/3204872568 learned_addr learned my addr 172.21.15.103:0/3204872568 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:39.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.004+0000 7f8728e4f700 1 -- 172.21.15.103:0/3204872568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872411fd70 con 0x7f872406d6e0 2023-07-02T09:02:39.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.004+0000 7f8728e4f700 1 --2- 172.21.15.103:0/3204872568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872406d6e0 0x7f87241201a0 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:39.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.004+0000 7f87217fa700 1 -- 172.21.15.103:0/3204872568 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8718004b80 con 0x7f872406d6e0 2023-07-02T09:02:39.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.004+0000 7f87217fa700 1 -- 172.21.15.103:0/3204872568 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8718004ce0 con 0x7f872406d6e0 2023-07-02T09:02:39.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.004+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8724121fc0 con 0x7f872406d6e0 2023-07-02T09:02:39.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.004+0000 7f87217fa700 1 -- 172.21.15.103:0/3204872568 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f871800c550 con 0x7f872406d6e0 2023-07-02T09:02:39.008 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.004+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87241228c0 con 0x7f872406d6e0 2023-07-02T09:02:39.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.005+0000 7f87217fa700 1 -- 172.21.15.103:0/3204872568 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f871800c6b0 con 0x7f872406d6e0 2023-07-02T09:02:39.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.005+0000 7f87217fa700 1 --2- 172.21.15.103:0/3204872568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f87080377c0 0x7f8708039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:39.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.005+0000 7f87217fa700 1 -- 172.21.15.103:0/3204872568 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f8718046a60 con 0x7f872406d6e0 2023-07-02T09:02:39.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.005+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8724122150 con 0x7f872406d6e0 2023-07-02T09:02:39.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.005+0000 7f8723fff700 1 --2- 172.21.15.103:0/3204872568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f87080377c0 0x7f8708039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:39.009 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.006+0000 7f8723fff700 1 --2- 172.21.15.103:0/3204872568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f87080377c0 0x7f8708039cd0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:39.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.010+0000 7f87217fa700 1 -- 172.21.15.103:0/3204872568 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f8718009c80 con 0x7f872406d6e0 2023-07-02T09:02:39.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.203+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8724061ea0 con 0x7f872406d6e0 2023-07-02T09:02:39.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.203+0000 7f87217fa700 1 -- 172.21.15.103:0/3204872568 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f87180098a0 con 0x7f872406d6e0 2023-07-02T09:02:39.205 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f87080377c0 msgr2=0x7f8708039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 --2- 172.21.15.103:0/3204872568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f87080377c0 0x7f8708039cd0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872406d6e0 msgr2=0x7f87241201a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 --2- 172.21.15.103:0/3204872568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872406d6e0 0x7f87241201a0 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 shutdown_connections 2023-07-02T09:02:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 --2- 172.21.15.103:0/3204872568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f87080377c0 0x7f8708039cd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 --2- 172.21.15.103:0/3204872568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f872406d6e0 0x7f87241201a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:39.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 >> 172.21.15.103:0/3204872568 conn(0x7f872409d810 msgr2=0x7f8724117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:39.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 shutdown_connections 2023-07-02T09:02:39.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:39.206+0000 7f872b0b3700 1 -- 172.21.15.103:0/3204872568 wait complete. 2023-07-02T09:02:39.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:40.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:40.363 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.359+0000 7f3887791700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:40.364 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.364+0000 7f3887791700 1 Processor -- start 2023-07-02T09:02:40.364 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.364+0000 7f3887791700 1 -- start start 2023-07-02T09:02:40.365 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3887791700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f388009a220 0x7f38801348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3887791700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388009cea0 con 0x7f388006d6e0 2023-07-02T09:02:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3887791700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388006d490 con 0x7f388009a220 2023-07-02T09:02:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3884d2c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f388009a220 0x7f38801348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3884d2c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f388009a220 0x7f38801348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48416/0 (socket says 172.21.15.103:48416) 2023-07-02T09:02:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f388552d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f388006d6e0 0x7f3880099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58788/0 (socket says 172.21.15.103:58788) 2023-07-02T09:02:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3884d2c700 1 -- 172.21.15.103:0/4131203144 learned_addr learned my addr 172.21.15.103:0/4131203144 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3884d2c700 1 -- 172.21.15.103:0/4131203144 >> v1:172.21.15.103:6789/0 conn(0x7f388006d6e0 legacy=0x7f3880099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3877fff700 1 -- 172.21.15.103:0/4131203144 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983132558 0 0) 0x7f388009cea0 con 0x7f388006d6e0 2023-07-02T09:02:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3884d2c700 1 -- 172.21.15.103:0/4131203144 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388006dc80 con 0x7f388009a220 2023-07-02T09:02:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3884d2c700 1 --2- 172.21.15.103:0/4131203144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388009a220 0x7f38801348e0 crc :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa2c9b4b248bf15 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3877fff700 1 -- 172.21.15.103:0/4131203144 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f387c010570 con 0x7f388009a220 2023-07-02T09:02:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3877fff700 1 -- 172.21.15.103:0/4131203144 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f387c0106d0 con 0x7f388009a220 2023-07-02T09:02:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.365+0000 7f3877fff700 1 -- 172.21.15.103:0/4131203144 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f387c010970 con 0x7f388009a220 2023-07-02T09:02:40.369 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.366+0000 7f3887791700 1 -- 172.21.15.103:0/4131203144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388009a220 msgr2=0x7f38801348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:40.369 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.366+0000 7f3887791700 1 --2- 172.21.15.103:0/4131203144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388009a220 0x7f38801348e0 crc :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:40.369 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.366+0000 7f3887791700 1 -- 172.21.15.103:0/4131203144 shutdown_connections 2023-07-02T09:02:40.369 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.366+0000 7f3887791700 1 --2- 172.21.15.103:0/4131203144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388009a220 0x7f38801348e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:40.369 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.366+0000 7f3887791700 1 -- 172.21.15.103:0/4131203144 >> 172.21.15.103:0/4131203144 conn(0x7f388009d810 msgr2=0x7f388009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.366+0000 7f3887791700 1 -- 172.21.15.103:0/4131203144 shutdown_connections 2023-07-02T09:02:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.366+0000 7f3887791700 1 -- 172.21.15.103:0/4131203144 wait complete. 2023-07-02T09:02:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.367+0000 7f3887791700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.367+0000 7f3887791700 1 Processor -- start 2023-07-02T09:02:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.367+0000 7f3887791700 1 -- start start 2023-07-02T09:02:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.368+0000 7f3887791700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388006d6e0 0x7f388011d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.368+0000 7f3887791700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880120a00 con 0x7f388006d6e0 2023-07-02T09:02:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.368+0000 7f388552d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388006d6e0 0x7f388011d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.368+0000 7f388552d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388006d6e0 0x7f388011d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48424/0 (socket says 172.21.15.103:48424) 2023-07-02T09:02:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.368+0000 7f388552d700 1 -- 172.21.15.103:0/732666494 learned_addr learned my addr 172.21.15.103:0/732666494 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.368+0000 7f388552d700 1 -- 172.21.15.103:0/732666494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388011c790 con 0x7f388006d6e0 2023-07-02T09:02:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.369+0000 7f388552d700 1 --2- 172.21.15.103:0/732666494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388006d6e0 0x7f388011d090 crc :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.369+0000 7f3875ffb700 1 -- 172.21.15.103:0/732666494 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3870004b80 con 0x7f388006d6e0 2023-07-02T09:02:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.369+0000 7f3875ffb700 1 -- 172.21.15.103:0/732666494 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3870004ce0 con 0x7f388006d6e0 2023-07-02T09:02:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.369+0000 7f3875ffb700 1 -- 172.21.15.103:0/732666494 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f387000c550 con 0x7f388006d6e0 2023-07-02T09:02:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.369+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388011cd00 con 0x7f388006d6e0 2023-07-02T09:02:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.369+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388011e3f0 con 0x7f388006d6e0 2023-07-02T09:02:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.370+0000 7f3875ffb700 1 -- 172.21.15.103:0/732666494 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f387000c6b0 con 0x7f388006d6e0 2023-07-02T09:02:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.370+0000 7f3875ffb700 1 --2- 172.21.15.103:0/732666494 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f38640377c0 0x7f3864039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.370+0000 7f3875ffb700 1 -- 172.21.15.103:0/732666494 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f38700469e0 con 0x7f388006d6e0 2023-07-02T09:02:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.370+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3880061ea0 con 0x7f388006d6e0 2023-07-02T09:02:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.370+0000 7f3884d2c700 1 --2- 172.21.15.103:0/732666494 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f38640377c0 0x7f3864039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.371+0000 7f3884d2c700 1 --2- 172.21.15.103:0/732666494 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f38640377c0 0x7f3864039cd0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:40.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.375+0000 7f3875ffb700 1 -- 172.21.15.103:0/732666494 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3870009c80 con 0x7f388006d6e0 2023-07-02T09:02:40.564 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.564+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3880073ca0 con 0x7f388006d6e0 2023-07-02T09:02:40.565 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.565+0000 7f3875ffb700 1 -- 172.21.15.103:0/732666494 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f38700098a0 con 0x7f388006d6e0 2023-07-02T09:02:40.565 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:40.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f38640377c0 msgr2=0x7f3864039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:40.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 --2- 172.21.15.103:0/732666494 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f38640377c0 0x7f3864039cd0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388006d6e0 msgr2=0x7f388011d090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 --2- 172.21.15.103:0/732666494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388006d6e0 0x7f388011d090 crc :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 shutdown_connections 2023-07-02T09:02:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 --2- 172.21.15.103:0/732666494 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f38640377c0 0x7f3864039cd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 --2- 172.21.15.103:0/732666494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f388006d6e0 0x7f388011d090 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 >> 172.21.15.103:0/732666494 conn(0x7f388009d810 msgr2=0x7f3880070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 shutdown_connections 2023-07-02T09:02:40.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:40.567+0000 7f3887791700 1 -- 172.21.15.103:0/732666494 wait complete. 2023-07-02T09:02:40.580 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:41.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:41.724 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.719+0000 7f06fdd23700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:41.725 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.725+0000 7f06fdd23700 1 Processor -- start 2023-07-02T09:02:41.725 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.725+0000 7f06fdd23700 1 -- start start 2023-07-02T09:02:41.726 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06fdd23700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f06f8132f50 0x7f06f8133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06fdd23700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f809e260 con 0x7f06f809c580 2023-07-02T09:02:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06fdd23700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80a1640 con 0x7f06f8132f50 2023-07-02T09:02:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f06f8132f50 0x7f06f8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f6ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f06f809c580 0x7f06f809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58790/0 (socket says 172.21.15.103:58790) 2023-07-02T09:02:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f06f8132f50 0x7f06f8133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48436/0 (socket says 172.21.15.103:48436) 2023-07-02T09:02:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f6ffd700 1 -- 172.21.15.103:0/1665653489 learned_addr learned my addr 172.21.15.103:0/1665653489 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f77fe700 1 -- 172.21.15.103:0/1665653489 >> v1:172.21.15.103:6789/0 conn(0x7f06f809c580 legacy=0x7f06f809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f77fe700 1 -- 172.21.15.103:0/1665653489 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8132a20 con 0x7f06f8132f50 2023-07-02T09:02:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f77fe700 1 --2- 172.21.15.103:0/1665653489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f8132f50 0x7f06f8133360 crc :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4419e8417157f3e3 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f67fc700 1 -- 172.21.15.103:0/1665653489 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f06e800e040 con 0x7f06f8132f50 2023-07-02T09:02:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f67fc700 1 -- 172.21.15.103:0/1665653489 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f06e8007670 con 0x7f06f8132f50 2023-07-02T09:02:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.726+0000 7f06f67fc700 1 -- 172.21.15.103:0/1665653489 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f06e8007910 con 0x7f06f8132f50 2023-07-02T09:02:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.727+0000 7f06fdd23700 1 -- 172.21.15.103:0/1665653489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f8132f50 msgr2=0x7f06f8133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.727+0000 7f06fdd23700 1 --2- 172.21.15.103:0/1665653489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f8132f50 0x7f06f8133360 crc :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.727+0000 7f06fdd23700 1 -- 172.21.15.103:0/1665653489 shutdown_connections 2023-07-02T09:02:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.727+0000 7f06fdd23700 1 --2- 172.21.15.103:0/1665653489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f8132f50 0x7f06f8133360 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.727+0000 7f06fdd23700 1 -- 172.21.15.103:0/1665653489 >> 172.21.15.103:0/1665653489 conn(0x7f06f80a2080 msgr2=0x7f06f80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.727+0000 7f06fdd23700 1 -- 172.21.15.103:0/1665653489 shutdown_connections 2023-07-02T09:02:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.727+0000 7f06fdd23700 1 -- 172.21.15.103:0/1665653489 wait complete. 2023-07-02T09:02:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.727+0000 7f06fdd23700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.728+0000 7f06fdd23700 1 Processor -- start 2023-07-02T09:02:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.728+0000 7f06fdd23700 1 -- start start 2023-07-02T09:02:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.729+0000 7f06fdd23700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f809c580 0x7f06f812b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.729+0000 7f06fdd23700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f806f9d0 con 0x7f06f809c580 2023-07-02T09:02:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.729+0000 7f06f77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f809c580 0x7f06f812b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.729+0000 7f06f77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f809c580 0x7f06f812b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48440/0 (socket says 172.21.15.103:48440) 2023-07-02T09:02:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.729+0000 7f06f77fe700 1 -- 172.21.15.103:0/951549207 learned_addr learned my addr 172.21.15.103:0/951549207 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.730+0000 7f06f77fe700 1 -- 172.21.15.103:0/951549207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8133a00 con 0x7f06f809c580 2023-07-02T09:02:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.730+0000 7f06f77fe700 1 --2- 172.21.15.103:0/951549207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f809c580 0x7f06f812b7b0 crc :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.730+0000 7f06dffff700 1 -- 172.21.15.103:0/951549207 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f06e800e040 con 0x7f06f809c580 2023-07-02T09:02:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.730+0000 7f06dffff700 1 -- 172.21.15.103:0/951549207 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f06e8013980 con 0x7f06f809c580 2023-07-02T09:02:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.730+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f806d160 con 0x7f06f809c580 2023-07-02T09:02:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.730+0000 7f06dffff700 1 -- 172.21.15.103:0/951549207 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f06e8013c20 con 0x7f06f809c580 2023-07-02T09:02:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.730+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f806da70 con 0x7f06f809c580 2023-07-02T09:02:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.731+0000 7f06dffff700 1 -- 172.21.15.103:0/951549207 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f06e8011040 con 0x7f06f809c580 2023-07-02T09:02:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.731+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8061ea0 con 0x7f06f809c580 2023-07-02T09:02:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.731+0000 7f06dffff700 1 --2- 172.21.15.103:0/951549207 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f06e40377c0 0x7f06e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.731+0000 7f06dffff700 1 -- 172.21.15.103:0/951549207 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f06e8048430 con 0x7f06f809c580 2023-07-02T09:02:41.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.731+0000 7f06f6ffd700 1 --2- 172.21.15.103:0/951549207 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f06e40377c0 0x7f06e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:41.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.732+0000 7f06f6ffd700 1 --2- 172.21.15.103:0/951549207 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f06e40377c0 0x7f06e4039cd0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:41.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.735+0000 7f06dffff700 1 -- 172.21.15.103:0/951549207 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f06e8010c50 con 0x7f06f809c580 2023-07-02T09:02:41.927 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.927+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f06f80725b0 con 0x7f06f809c580 2023-07-02T09:02:41.927 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.927+0000 7f06dffff700 1 -- 172.21.15.103:0/951549207 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f06e8010a70 con 0x7f06f809c580 2023-07-02T09:02:41.928 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:41.929 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f06e40377c0 msgr2=0x7f06e4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 --2- 172.21.15.103:0/951549207 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f06e40377c0 0x7f06e4039cd0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f809c580 msgr2=0x7f06f812b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 --2- 172.21.15.103:0/951549207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f809c580 0x7f06f812b7b0 crc :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 shutdown_connections 2023-07-02T09:02:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 --2- 172.21.15.103:0/951549207 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f06e40377c0 0x7f06e4039cd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 --2- 172.21.15.103:0/951549207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f809c580 0x7f06f812b7b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:41.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 >> 172.21.15.103:0/951549207 conn(0x7f06f80a2080 msgr2=0x7f06f81babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:41.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 shutdown_connections 2023-07-02T09:02:41.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:41.929+0000 7f06fdd23700 1 -- 172.21.15.103:0/951549207 wait complete. 2023-07-02T09:02:41.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:42.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~176s 2023-07-02T09:02:42.943 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:43.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.082+0000 7f66e672e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:43.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.087+0000 7f66e672e700 1 Processor -- start 2023-07-02T09:02:43.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.087+0000 7f66e672e700 1 -- start start 2023-07-02T09:02:43.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66e672e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f66e0133360 0x7f66e009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:43.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66e672e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e009e260 con 0x7f66e0132f50 2023-07-02T09:02:43.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66e672e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e00a1640 con 0x7f66e0133360 2023-07-02T09:02:43.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66df7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f66e0133360 0x7f66e009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:43.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66df7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f66e0133360 0x7f66e009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48450/0 (socket says 172.21.15.103:48450) 2023-07-02T09:02:43.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66dffff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f66e0132f50 0x7f66e009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:58794/0 (socket says 172.21.15.103:58794) 2023-07-02T09:02:43.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66df7fe700 1 -- 172.21.15.103:0/3824097094 learned_addr learned my addr 172.21.15.103:0/3824097094 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:43.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66df7fe700 1 -- 172.21.15.103:0/3824097094 >> v1:172.21.15.103:6789/0 conn(0x7f66e0132f50 legacy=0x7f66e009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:43.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66df7fe700 1 -- 172.21.15.103:0/3824097094 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e0132a20 con 0x7f66e0133360 2023-07-02T09:02:43.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66deffd700 1 -- 172.21.15.103:0/3824097094 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500582981 0 0) 0x7f66e009e260 con 0x7f66e0132f50 2023-07-02T09:02:43.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66df7fe700 1 --2- 172.21.15.103:0/3824097094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0133360 0x7f66e009cc90 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeaf30542462bb3f server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:43.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66deffd700 1 -- 172.21.15.103:0/3824097094 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f66c8009d90 con 0x7f66e0133360 2023-07-02T09:02:43.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66deffd700 1 -- 172.21.15.103:0/3824097094 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f66c800e230 con 0x7f66e0133360 2023-07-02T09:02:43.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.088+0000 7f66deffd700 1 -- 172.21.15.103:0/3824097094 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f66c800e4d0 con 0x7f66e0133360 2023-07-02T09:02:43.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.089+0000 7f66e672e700 1 -- 172.21.15.103:0/3824097094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0133360 msgr2=0x7f66e009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:43.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.089+0000 7f66e672e700 1 --2- 172.21.15.103:0/3824097094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0133360 0x7f66e009cc90 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:43.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.089+0000 7f66e672e700 1 -- 172.21.15.103:0/3824097094 shutdown_connections 2023-07-02T09:02:43.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.089+0000 7f66e672e700 1 --2- 172.21.15.103:0/3824097094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0133360 0x7f66e009cc90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:43.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.089+0000 7f66e672e700 1 -- 172.21.15.103:0/3824097094 >> 172.21.15.103:0/3824097094 conn(0x7f66e00a2080 msgr2=0x7f66e00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:43.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.089+0000 7f66e672e700 1 -- 172.21.15.103:0/3824097094 shutdown_connections 2023-07-02T09:02:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.089+0000 7f66e672e700 1 -- 172.21.15.103:0/3824097094 wait complete. 2023-07-02T09:02:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.089+0000 7f66e672e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.090+0000 7f66e672e700 1 Processor -- start 2023-07-02T09:02:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.091+0000 7f66e672e700 1 -- start start 2023-07-02T09:02:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.091+0000 7f66e672e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0132f50 0x7f66e01ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.091+0000 7f66e672e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e006d370 con 0x7f66e0132f50 2023-07-02T09:02:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.091+0000 7f66dffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0132f50 0x7f66e01ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:43.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.091+0000 7f66dffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0132f50 0x7f66e01ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48458/0 (socket says 172.21.15.103:48458) 2023-07-02T09:02:43.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.091+0000 7f66dffff700 1 -- 172.21.15.103:0/1561811413 learned_addr learned my addr 172.21.15.103:0/1561811413 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:43.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.092+0000 7f66dffff700 1 -- 172.21.15.103:0/1561811413 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e006dde0 con 0x7f66e0132f50 2023-07-02T09:02:43.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.092+0000 7f66dffff700 1 --2- 172.21.15.103:0/1561811413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0132f50 0x7f66e01ba9b0 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.092+0000 7f66dcff9700 1 -- 172.21.15.103:0/1561811413 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f66d0004b80 con 0x7f66e0132f50 2023-07-02T09:02:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.092+0000 7f66dcff9700 1 -- 172.21.15.103:0/1561811413 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f66d0004ce0 con 0x7f66e0132f50 2023-07-02T09:02:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.092+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e01ba4f0 con 0x7f66e0132f50 2023-07-02T09:02:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.092+0000 7f66dcff9700 1 -- 172.21.15.103:0/1561811413 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f66d000c4f0 con 0x7f66e0132f50 2023-07-02T09:02:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.092+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e0071e40 con 0x7f66e0132f50 2023-07-02T09:02:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.093+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e0061ea0 con 0x7f66e0132f50 2023-07-02T09:02:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.093+0000 7f66dcff9700 1 -- 172.21.15.103:0/1561811413 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f66d000c650 con 0x7f66e0132f50 2023-07-02T09:02:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.093+0000 7f66dcff9700 1 --2- 172.21.15.103:0/1561811413 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f66c00377c0 0x7f66c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.093+0000 7f66dcff9700 1 -- 172.21.15.103:0/1561811413 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f66d0046970 con 0x7f66e0132f50 2023-07-02T09:02:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.094+0000 7f66df7fe700 1 --2- 172.21.15.103:0/1561811413 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f66c00377c0 0x7f66c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.094+0000 7f66df7fe700 1 --2- 172.21.15.103:0/1561811413 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f66c00377c0 0x7f66c0039cd0 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:43.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.098+0000 7f66dcff9700 1 -- 172.21.15.103:0/1561811413 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f66d0009c80 con 0x7f66e0132f50 2023-07-02T09:02:43.288 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.288+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f66e00725b0 con 0x7f66e0132f50 2023-07-02T09:02:43.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.288+0000 7f66dcff9700 1 -- 172.21.15.103:0/1561811413 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f66d00098a0 con 0x7f66e0132f50 2023-07-02T09:02:43.289 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:43.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.290+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f66c00377c0 msgr2=0x7f66c0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:43.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.290+0000 7f66e672e700 1 --2- 172.21.15.103:0/1561811413 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f66c00377c0 0x7f66c0039cd0 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:43.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.290+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0132f50 msgr2=0x7f66e01ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:43.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.290+0000 7f66e672e700 1 --2- 172.21.15.103:0/1561811413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0132f50 0x7f66e01ba9b0 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:43.291 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.290+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 shutdown_connections 2023-07-02T09:02:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.290+0000 7f66e672e700 1 --2- 172.21.15.103:0/1561811413 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f66c00377c0 0x7f66c0039cd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.290+0000 7f66e672e700 1 --2- 172.21.15.103:0/1561811413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0132f50 0x7f66e01ba9b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.291+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 >> 172.21.15.103:0/1561811413 conn(0x7f66e00a2080 msgr2=0x7f66e00a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.291+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 shutdown_connections 2023-07-02T09:02:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:43.291+0000 7f66e672e700 1 -- 172.21.15.103:0/1561811413 wait complete. 2023-07-02T09:02:43.304 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:44.304 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:44.448 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.444+0000 7f83103ea700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:44.449 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.449+0000 7f83103ea700 1 Processor -- start 2023-07-02T09:02:44.449 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.449+0000 7f83103ea700 1 -- start start 2023-07-02T09:02:44.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f83103ea700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8308133360 0x7f830809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:44.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f83103ea700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830809e260 con 0x7f8308132f50 2023-07-02T09:02:44.450 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f83103ea700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83080a1640 con 0x7f8308133360 2023-07-02T09:02:44.451 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f830d985700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8308133360 0x7f830809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:44.451 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f830e186700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f8308132f50 0x7f830809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41124/0 (socket says 172.21.15.103:41124) 2023-07-02T09:02:44.451 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f830e186700 1 -- 172.21.15.103:0/4178440343 learned_addr learned my addr 172.21.15.103:0/4178440343 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:44.451 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f830d985700 1 --2- 172.21.15.103:0/4178440343 >> v2:172.21.15.103:3300/0 conn(0x7f8308133360 0x7f830809cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57610/0 (socket says 172.21.15.103:57610) 2023-07-02T09:02:44.452 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f830d985700 1 -- 172.21.15.103:0/4178440343 >> v1:172.21.15.103:6789/0 conn(0x7f8308132f50 legacy=0x7f830809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:44.452 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f830d985700 1 -- 172.21.15.103:0/4178440343 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8308132a20 con 0x7f8308133360 2023-07-02T09:02:44.453 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.450+0000 7f830d985700 1 --2- 172.21.15.103:0/4178440343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308133360 0x7f830809cc90 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802aa8bae0d05f server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:44.453 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f830d184700 1 -- 172.21.15.103:0/4178440343 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8304009cd0 con 0x7f8308133360 2023-07-02T09:02:44.453 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f830d184700 1 -- 172.21.15.103:0/4178440343 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8304009e30 con 0x7f8308133360 2023-07-02T09:02:44.453 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f830d184700 1 -- 172.21.15.103:0/4178440343 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f830400e360 con 0x7f8308133360 2023-07-02T09:02:44.453 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f83103ea700 1 -- 172.21.15.103:0/4178440343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308133360 msgr2=0x7f830809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:44.453 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f83103ea700 1 --2- 172.21.15.103:0/4178440343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308133360 0x7f830809cc90 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:44.453 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f83103ea700 1 -- 172.21.15.103:0/4178440343 shutdown_connections 2023-07-02T09:02:44.454 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f83103ea700 1 --2- 172.21.15.103:0/4178440343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308133360 0x7f830809cc90 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:44.454 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f83103ea700 1 -- 172.21.15.103:0/4178440343 >> 172.21.15.103:0/4178440343 conn(0x7f83080a2080 msgr2=0x7f83080a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:44.454 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f83103ea700 1 -- 172.21.15.103:0/4178440343 shutdown_connections 2023-07-02T09:02:44.454 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.451+0000 7f83103ea700 1 -- 172.21.15.103:0/4178440343 wait complete. 2023-07-02T09:02:44.454 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.452+0000 7f83103ea700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:44.456 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.453+0000 7f83103ea700 1 Processor -- start 2023-07-02T09:02:44.456 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.453+0000 7f83103ea700 1 -- start start 2023-07-02T09:02:44.456 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.453+0000 7f83103ea700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308132f50 0x7f830812bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:44.456 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.453+0000 7f83103ea700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830806d260 con 0x7f8308132f50 2023-07-02T09:02:44.456 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.453+0000 7f830e186700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308132f50 0x7f830812bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.453+0000 7f830e186700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308132f50 0x7f830812bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57616/0 (socket says 172.21.15.103:57616) 2023-07-02T09:02:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.453+0000 7f830e186700 1 -- 172.21.15.103:0/1193045274 learned_addr learned my addr 172.21.15.103:0/1193045274 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.454+0000 7f830e186700 1 -- 172.21.15.103:0/1193045274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830812bb90 con 0x7f8308132f50 2023-07-02T09:02:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.454+0000 7f830e186700 1 --2- 172.21.15.103:0/1193045274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308132f50 0x7f830812bea0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.454+0000 7f82feffd700 1 -- 172.21.15.103:0/1193045274 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f82f8005a60 con 0x7f8308132f50 2023-07-02T09:02:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.454+0000 7f82feffd700 1 -- 172.21.15.103:0/1193045274 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f82f8005bc0 con 0x7f8308132f50 2023-07-02T09:02:44.457 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.454+0000 7f82feffd700 1 -- 172.21.15.103:0/1193045274 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f82f8005e60 con 0x7f8308132f50 2023-07-02T09:02:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.454+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830806f730 con 0x7f8308132f50 2023-07-02T09:02:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.454+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830806f490 con 0x7f8308132f50 2023-07-02T09:02:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.455+0000 7f82feffd700 1 -- 172.21.15.103:0/1193045274 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f82f80070b0 con 0x7f8308132f50 2023-07-02T09:02:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.455+0000 7f82feffd700 1 --2- 172.21.15.103:0/1193045274 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82ec0377c0 0x7f82ec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.455+0000 7f82feffd700 1 -- 172.21.15.103:0/1193045274 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f82f80469f0 con 0x7f8308132f50 2023-07-02T09:02:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.455+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830806f490 con 0x7f8308132f50 2023-07-02T09:02:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.455+0000 7f830d985700 1 --2- 172.21.15.103:0/1193045274 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82ec0377c0 0x7f82ec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:44.458 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.456+0000 7f830d985700 1 --2- 172.21.15.103:0/1193045274 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82ec0377c0 0x7f82ec039cd0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:44.460 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.460+0000 7f82feffd700 1 -- 172.21.15.103:0/1193045274 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f830806f490 con 0x7f8308132f50 2023-07-02T09:02:44.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.651+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f830806f490 con 0x7f8308132f50 2023-07-02T09:02:44.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.651+0000 7f82feffd700 1 -- 172.21.15.103:0/1193045274 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f830806f490 con 0x7f8308132f50 2023-07-02T09:02:44.652 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:44.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82ec0377c0 msgr2=0x7f82ec039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:44.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 --2- 172.21.15.103:0/1193045274 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82ec0377c0 0x7f82ec039cd0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:44.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308132f50 msgr2=0x7f830812bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:44.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 --2- 172.21.15.103:0/1193045274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308132f50 0x7f830812bea0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:44.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 shutdown_connections 2023-07-02T09:02:44.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 --2- 172.21.15.103:0/1193045274 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f82ec0377c0 0x7f82ec039cd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:44.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 --2- 172.21.15.103:0/1193045274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308132f50 0x7f830812bea0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:44.654 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 >> 172.21.15.103:0/1193045274 conn(0x7f83080a2080 msgr2=0x7f83081bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:44.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 shutdown_connections 2023-07-02T09:02:44.655 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:44.653+0000 7f83103ea700 1 -- 172.21.15.103:0/1193045274 wait complete. 2023-07-02T09:02:44.666 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:45.667 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:45.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.807+0000 7f7b56748700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:45.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.813+0000 7f7b56748700 1 Processor -- start 2023-07-02T09:02:45.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.813+0000 7f7b56748700 1 -- start start 2023-07-02T09:02:45.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b56748700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b50132f50 0x7f7b50133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:45.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b56748700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5009e260 con 0x7f7b5009c580 2023-07-02T09:02:45.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b56748700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b500a1640 con 0x7f7b50132f50 2023-07-02T09:02:45.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b4ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b50132f50 0x7f7b50133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:45.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b4f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7b5009c580 0x7f7b5009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41134/0 (socket says 172.21.15.103:41134) 2023-07-02T09:02:45.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b4ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b50132f50 0x7f7b50133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57618/0 (socket says 172.21.15.103:57618) 2023-07-02T09:02:45.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b4f7fe700 1 -- 172.21.15.103:0/3926854726 learned_addr learned my addr 172.21.15.103:0/3926854726 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:45.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b4ffff700 1 -- 172.21.15.103:0/3926854726 >> v1:172.21.15.103:6789/0 conn(0x7f7b5009c580 legacy=0x7f7b5009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:45.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b4ffff700 1 -- 172.21.15.103:0/3926854726 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50132a20 con 0x7f7b50132f50 2023-07-02T09:02:45.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.814+0000 7f7b4ffff700 1 --2- 172.21.15.103:0/3926854726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b50132f50 0x7f7b50133360 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34974b3e9f3d74cf server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:45.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.816+0000 7f7b4effd700 1 -- 172.21.15.103:0/3926854726 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7b40009cd0 con 0x7f7b50132f50 2023-07-02T09:02:45.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.816+0000 7f7b4effd700 1 -- 172.21.15.103:0/3926854726 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7b40009e30 con 0x7f7b50132f50 2023-07-02T09:02:45.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.816+0000 7f7b4effd700 1 -- 172.21.15.103:0/3926854726 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7b4000e360 con 0x7f7b50132f50 2023-07-02T09:02:45.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.817+0000 7f7b56748700 1 -- 172.21.15.103:0/3926854726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b50132f50 msgr2=0x7f7b50133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:45.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.817+0000 7f7b56748700 1 --2- 172.21.15.103:0/3926854726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b50132f50 0x7f7b50133360 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:45.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.817+0000 7f7b56748700 1 -- 172.21.15.103:0/3926854726 shutdown_connections 2023-07-02T09:02:45.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.817+0000 7f7b56748700 1 --2- 172.21.15.103:0/3926854726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b50132f50 0x7f7b50133360 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:45.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.817+0000 7f7b56748700 1 -- 172.21.15.103:0/3926854726 >> 172.21.15.103:0/3926854726 conn(0x7f7b500a2080 msgr2=0x7f7b500a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:45.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.817+0000 7f7b56748700 1 -- 172.21.15.103:0/3926854726 shutdown_connections 2023-07-02T09:02:45.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.817+0000 7f7b56748700 1 -- 172.21.15.103:0/3926854726 wait complete. 2023-07-02T09:02:45.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.817+0000 7f7b56748700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:45.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.818+0000 7f7b56748700 1 Processor -- start 2023-07-02T09:02:45.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.818+0000 7f7b56748700 1 -- start start 2023-07-02T09:02:45.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b56748700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b5009c580 0x7f7b5012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:45.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b56748700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5006f9d0 con 0x7f7b5009c580 2023-07-02T09:02:45.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b4ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b5009c580 0x7f7b5012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:45.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b4ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b5009c580 0x7f7b5012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57634/0 (socket says 172.21.15.103:57634) 2023-07-02T09:02:45.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b4ffff700 1 -- 172.21.15.103:0/253916516 learned_addr learned my addr 172.21.15.103:0/253916516 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b4ffff700 1 -- 172.21.15.103:0/253916516 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50133a00 con 0x7f7b5009c580 2023-07-02T09:02:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b4ffff700 1 --2- 172.21.15.103:0/253916516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b5009c580 0x7f7b5012b7b0 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b4cff9700 1 -- 172.21.15.103:0/253916516 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7b40013530 con 0x7f7b5009c580 2023-07-02T09:02:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b4cff9700 1 -- 172.21.15.103:0/253916516 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7b40013690 con 0x7f7b5009c580 2023-07-02T09:02:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5006d160 con 0x7f7b5009c580 2023-07-02T09:02:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.819+0000 7f7b4cff9700 1 -- 172.21.15.103:0/253916516 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7b40013930 con 0x7f7b5009c580 2023-07-02T09:02:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.820+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5006da70 con 0x7f7b5009c580 2023-07-02T09:02:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.820+0000 7f7b4cff9700 1 -- 172.21.15.103:0/253916516 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7b40013ae0 con 0x7f7b5009c580 2023-07-02T09:02:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.820+0000 7f7b4cff9700 1 --2- 172.21.15.103:0/253916516 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b3c0377c0 0x7f7b3c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.820+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b50061ea0 con 0x7f7b5009c580 2023-07-02T09:02:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.820+0000 7f7b4cff9700 1 -- 172.21.15.103:0/253916516 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7b40018040 con 0x7f7b5009c580 2023-07-02T09:02:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.821+0000 7f7b4f7fe700 1 --2- 172.21.15.103:0/253916516 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b3c0377c0 0x7f7b3c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:45.823 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.821+0000 7f7b4f7fe700 1 --2- 172.21.15.103:0/253916516 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b3c0377c0 0x7f7b3c039cd0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:45.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:45.825+0000 7f7b4cff9700 1 -- 172.21.15.103:0/253916516 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7b40018e00 con 0x7f7b5009c580 2023-07-02T09:02:46.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.016+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7b500725b0 con 0x7f7b5009c580 2023-07-02T09:02:46.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.017+0000 7f7b4cff9700 1 -- 172.21.15.103:0/253916516 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7b40010e60 con 0x7f7b5009c580 2023-07-02T09:02:46.018 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:46.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b3c0377c0 msgr2=0x7f7b3c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:46.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 --2- 172.21.15.103:0/253916516 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b3c0377c0 0x7f7b3c039cd0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:46.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b5009c580 msgr2=0x7f7b5012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:46.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 --2- 172.21.15.103:0/253916516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b5009c580 0x7f7b5012b7b0 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:46.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 shutdown_connections 2023-07-02T09:02:46.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 --2- 172.21.15.103:0/253916516 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b3c0377c0 0x7f7b3c039cd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:46.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 --2- 172.21.15.103:0/253916516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b5009c580 0x7f7b5012b7b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:46.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 >> 172.21.15.103:0/253916516 conn(0x7f7b500a2080 msgr2=0x7f7b501babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:46.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 shutdown_connections 2023-07-02T09:02:46.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:46.019+0000 7f7b56748700 1 -- 172.21.15.103:0/253916516 wait complete. 2023-07-02T09:02:46.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:47.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:47.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.173+0000 7f1cd2d60700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:47.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.178+0000 7f1cd2d60700 1 Processor -- start 2023-07-02T09:02:47.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.178+0000 7f1cd2d60700 1 -- start start 2023-07-02T09:02:47.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1cd2d60700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ccc132030 0x7f1ccc134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:47.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1cd2d60700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ccc06df60 con 0x7f1ccc12f250 2023-07-02T09:02:47.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1cd2d60700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ccc09a710 con 0x7f1ccc132030 2023-07-02T09:02:47.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccbfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ccc132030 0x7f1ccc134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:47.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1cd0afc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1ccc12f250 0x7f1ccc12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41138/0 (socket says 172.21.15.103:41138) 2023-07-02T09:02:47.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccbfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ccc132030 0x7f1ccc134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57648/0 (socket says 172.21.15.103:57648) 2023-07-02T09:02:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1cd0afc700 1 -- 172.21.15.103:0/3431303020 learned_addr learned my addr 172.21.15.103:0/3431303020 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccbfff700 1 -- 172.21.15.103:0/3431303020 >> v1:172.21.15.103:6789/0 conn(0x7f1ccc12f250 legacy=0x7f1ccc12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccbfff700 1 -- 172.21.15.103:0/3431303020 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc12ed20 con 0x7f1ccc132030 2023-07-02T09:02:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccb7fe700 1 -- 172.21.15.103:0/3431303020 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961488881 0 0) 0x7f1ccc06df60 con 0x7f1ccc12f250 2023-07-02T09:02:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccbfff700 1 --2- 172.21.15.103:0/3431303020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc132030 0x7f1ccc134450 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7fd635a7950047 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccb7fe700 1 -- 172.21.15.103:0/3431303020 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1cb8010440 con 0x7f1ccc132030 2023-07-02T09:02:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccb7fe700 1 -- 172.21.15.103:0/3431303020 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1cb80105a0 con 0x7f1ccc132030 2023-07-02T09:02:47.182 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.179+0000 7f1ccb7fe700 1 -- 172.21.15.103:0/3431303020 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1cb8010840 con 0x7f1ccc132030 2023-07-02T09:02:47.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.180+0000 7f1cd2d60700 1 -- 172.21.15.103:0/3431303020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc132030 msgr2=0x7f1ccc134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:47.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.180+0000 7f1cd2d60700 1 --2- 172.21.15.103:0/3431303020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc132030 0x7f1ccc134450 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:47.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.180+0000 7f1cd2d60700 1 -- 172.21.15.103:0/3431303020 shutdown_connections 2023-07-02T09:02:47.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.180+0000 7f1cd2d60700 1 --2- 172.21.15.103:0/3431303020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc132030 0x7f1ccc134450 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:47.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.180+0000 7f1cd2d60700 1 -- 172.21.15.103:0/3431303020 >> 172.21.15.103:0/3431303020 conn(0x7f1ccc09d810 msgr2=0x7f1ccc09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:47.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.180+0000 7f1cd2d60700 1 -- 172.21.15.103:0/3431303020 shutdown_connections 2023-07-02T09:02:47.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.180+0000 7f1cd2d60700 1 -- 172.21.15.103:0/3431303020 wait complete. 2023-07-02T09:02:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.180+0000 7f1cd2d60700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.181+0000 7f1cd2d60700 1 Processor -- start 2023-07-02T09:02:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.181+0000 7f1cd2d60700 1 -- start start 2023-07-02T09:02:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.182+0000 7f1cd2d60700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc12f250 0x7f1ccc06fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.182+0000 7f1cd2d60700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc071390 con 0x7f1ccc12f250 2023-07-02T09:02:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.182+0000 7f1cd0afc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc12f250 0x7f1ccc06fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:47.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.182+0000 7f1cd0afc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc12f250 0x7f1ccc06fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57664/0 (socket says 172.21.15.103:57664) 2023-07-02T09:02:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.182+0000 7f1cd0afc700 1 -- 172.21.15.103:0/1474576706 learned_addr learned my addr 172.21.15.103:0/1474576706 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.182+0000 7f1cd0afc700 1 -- 172.21.15.103:0/1474576706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc1292e0 con 0x7f1ccc12f250 2023-07-02T09:02:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.183+0000 7f1cd0afc700 1 --2- 172.21.15.103:0/1474576706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc12f250 0x7f1ccc06fe70 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.183+0000 7f1cc97fa700 1 -- 172.21.15.103:0/1474576706 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1cc0004b80 con 0x7f1ccc12f250 2023-07-02T09:02:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.183+0000 7f1cc97fa700 1 -- 172.21.15.103:0/1474576706 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1cc0004ce0 con 0x7f1ccc12f250 2023-07-02T09:02:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.183+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ccc128f40 con 0x7f1ccc12f250 2023-07-02T09:02:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.183+0000 7f1cc97fa700 1 -- 172.21.15.103:0/1474576706 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1cc000c4f0 con 0x7f1ccc12f250 2023-07-02T09:02:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.183+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ccc1179c0 con 0x7f1ccc12f250 2023-07-02T09:02:47.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.183+0000 7f1cc97fa700 1 -- 172.21.15.103:0/1474576706 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f1cc000c650 con 0x7f1ccc12f250 2023-07-02T09:02:47.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.184+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ccc061ea0 con 0x7f1ccc12f250 2023-07-02T09:02:47.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.184+0000 7f1cc97fa700 1 --2- 172.21.15.103:0/1474576706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cb00377c0 0x7f1cb0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:47.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.184+0000 7f1cc97fa700 1 -- 172.21.15.103:0/1474576706 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f1cc00467f0 con 0x7f1ccc12f250 2023-07-02T09:02:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.184+0000 7f1ccbfff700 1 --2- 172.21.15.103:0/1474576706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cb00377c0 0x7f1cb0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.184+0000 7f1ccbfff700 1 --2- 172.21.15.103:0/1474576706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cb00377c0 0x7f1cb0039cd0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:47.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.188+0000 7f1cc97fa700 1 -- 172.21.15.103:0/1474576706 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f1cc0009ae0 con 0x7f1ccc12f250 2023-07-02T09:02:47.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.378+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1ccc1292c0 con 0x7f1ccc12f250 2023-07-02T09:02:47.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.379+0000 7f1cc97fa700 1 -- 172.21.15.103:0/1474576706 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f1cc00090b0 con 0x7f1ccc12f250 2023-07-02T09:02:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:47.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cb00377c0 msgr2=0x7f1cb0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:47.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 --2- 172.21.15.103:0/1474576706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cb00377c0 0x7f1cb0039cd0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:47.381 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc12f250 msgr2=0x7f1ccc06fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:47.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 --2- 172.21.15.103:0/1474576706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc12f250 0x7f1ccc06fe70 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:47.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 shutdown_connections 2023-07-02T09:02:47.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 --2- 172.21.15.103:0/1474576706 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cb00377c0 0x7f1cb0039cd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:47.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 --2- 172.21.15.103:0/1474576706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ccc12f250 0x7f1ccc06fe70 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:47.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 >> 172.21.15.103:0/1474576706 conn(0x7f1ccc09d810 msgr2=0x7f1ccc12dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:47.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 shutdown_connections 2023-07-02T09:02:47.382 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:47.381+0000 7f1cd2d60700 1 -- 172.21.15.103:0/1474576706 wait complete. 2023-07-02T09:02:47.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:47.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~182s 2023-07-02T09:02:48.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:48.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.536+0000 7f9a1bf0a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:48.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.541+0000 7f9a1bf0a700 1 Processor -- start 2023-07-02T09:02:48.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.541+0000 7f9a1bf0a700 1 -- start start 2023-07-02T09:02:48.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.542+0000 7f9a1bf0a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9a14098a00 0x7f9a14098e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:48.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.542+0000 7f9a1bf0a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1406f8b0 con 0x7f9a14099350 2023-07-02T09:02:48.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.542+0000 7f9a1bf0a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a14072510 con 0x7f9a14098a00 2023-07-02T09:02:48.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.542+0000 7f9a19ca6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9a14098a00 0x7f9a14098e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:48.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.542+0000 7f9a194a5700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9a14099350 0x7f9a1412d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41150/0 (socket says 172.21.15.103:41150) 2023-07-02T09:02:48.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.542+0000 7f9a194a5700 1 -- 172.21.15.103:0/4134730429 learned_addr learned my addr 172.21.15.103:0/4134730429 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.542+0000 7f9a19ca6700 1 -- 172.21.15.103:0/4134730429 >> v1:172.21.15.103:6789/0 conn(0x7f9a14099350 legacy=0x7f9a1412d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.542+0000 7f9a19ca6700 1 -- 172.21.15.103:0/4134730429 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1409aec0 con 0x7f9a14098a00 2023-07-02T09:02:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a19ca6700 1 --2- 172.21.15.103:0/4134730429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a14098e10 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761e22f3109a34a5 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a18ca4700 1 -- 172.21.15.103:0/4134730429 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9a04009cd0 con 0x7f9a14098a00 2023-07-02T09:02:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a18ca4700 1 -- 172.21.15.103:0/4134730429 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9a04009e30 con 0x7f9a14098a00 2023-07-02T09:02:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a18ca4700 1 -- 172.21.15.103:0/4134730429 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9a0400e360 con 0x7f9a14098a00 2023-07-02T09:02:48.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/4134730429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 msgr2=0x7f9a14098e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:48.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a1bf0a700 1 --2- 172.21.15.103:0/4134730429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a14098e10 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:48.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/4134730429 shutdown_connections 2023-07-02T09:02:48.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a1bf0a700 1 --2- 172.21.15.103:0/4134730429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a14098e10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:48.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/4134730429 >> 172.21.15.103:0/4134730429 conn(0x7f9a1409d810 msgr2=0x7f9a1409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:48.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.543+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/4134730429 shutdown_connections 2023-07-02T09:02:48.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.544+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/4134730429 wait complete. 2023-07-02T09:02:48.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.544+0000 7f9a1bf0a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:48.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.545+0000 7f9a1bf0a700 1 Processor -- start 2023-07-02T09:02:48.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.545+0000 7f9a1bf0a700 1 -- start start 2023-07-02T09:02:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.545+0000 7f9a1bf0a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a141c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:48.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.545+0000 7f9a1bf0a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1411cc50 con 0x7f9a14098a00 2023-07-02T09:02:48.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.545+0000 7f9a19ca6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a141c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:48.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.545+0000 7f9a19ca6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a141c4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57694/0 (socket says 172.21.15.103:57694) 2023-07-02T09:02:48.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.545+0000 7f9a19ca6700 1 -- 172.21.15.103:0/108905552 learned_addr learned my addr 172.21.15.103:0/108905552 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:48.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.546+0000 7f9a19ca6700 1 -- 172.21.15.103:0/108905552 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1411ca20 con 0x7f9a14098a00 2023-07-02T09:02:48.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.546+0000 7f9a19ca6700 1 --2- 172.21.15.103:0/108905552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a141c4ea0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:48.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.546+0000 7f9a0a7fc700 1 -- 172.21.15.103:0/108905552 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9a04013740 con 0x7f9a14098a00 2023-07-02T09:02:48.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.546+0000 7f9a0a7fc700 1 -- 172.21.15.103:0/108905552 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9a040138a0 con 0x7f9a14098a00 2023-07-02T09:02:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.546+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a141b9bd0 con 0x7f9a14098a00 2023-07-02T09:02:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.546+0000 7f9a0a7fc700 1 -- 172.21.15.103:0/108905552 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9a04013b40 con 0x7f9a14098a00 2023-07-02T09:02:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.546+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1411d3a0 con 0x7f9a14098a00 2023-07-02T09:02:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.547+0000 7f9a0a7fc700 1 -- 172.21.15.103:0/108905552 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f9a04013cf0 con 0x7f9a14098a00 2023-07-02T09:02:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.547+0000 7f9a0a7fc700 1 --2- 172.21.15.103:0/108905552 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.547+0000 7f9a0a7fc700 1 -- 172.21.15.103:0/108905552 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9a04018040 con 0x7f9a14098a00 2023-07-02T09:02:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.547+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1411d920 con 0x7f9a14098a00 2023-07-02T09:02:48.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.547+0000 7f9a194a5700 1 --2- 172.21.15.103:0/108905552 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:48.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.548+0000 7f9a194a5700 1 --2- 172.21.15.103:0/108905552 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:48.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.552+0000 7f9a0a7fc700 1 -- 172.21.15.103:0/108905552 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f9a04011380 con 0x7f9a14098a00 2023-07-02T09:02:48.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.740+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9a14061ea0 con 0x7f9a14098a00 2023-07-02T09:02:48.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.741+0000 7f9a0a7fc700 1 -- 172.21.15.103:0/108905552 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f9a0400f2a0 con 0x7f9a14098a00 2023-07-02T09:02:48.741 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:48.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 msgr2=0x7f99f8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:48.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 --2- 172.21.15.103:0/108905552 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:48.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 msgr2=0x7f9a141c4ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:48.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 --2- 172.21.15.103:0/108905552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a141c4ea0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:48.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 shutdown_connections 2023-07-02T09:02:48.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 --2- 172.21.15.103:0/108905552 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f99f80377c0 0x7f99f8039cd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:48.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 --2- 172.21.15.103:0/108905552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a14098a00 0x7f9a141c4ea0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:48.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 >> 172.21.15.103:0/108905552 conn(0x7f9a1409d810 msgr2=0x7f9a141b8790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:48.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 shutdown_connections 2023-07-02T09:02:48.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:48.743+0000 7f9a1bf0a700 1 -- 172.21.15.103:0/108905552 wait complete. 2023-07-02T09:02:48.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:49.756 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:49.901 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.897+0000 7fd04b885700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:49.902 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.902+0000 7fd04b885700 1 Processor -- start 2023-07-02T09:02:49.902 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.902+0000 7fd04b885700 1 -- start start 2023-07-02T09:02:49.903 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd04b885700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd04406d6e0 0x7fd044099b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:49.903 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd04b885700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04409cea0 con 0x7fd04409a050 2023-07-02T09:02:49.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd04b885700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04406d490 con 0x7fd04406d6e0 2023-07-02T09:02:49.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd049621700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd04406d6e0 0x7fd044099b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:49.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd048e20700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd04409a050 0x7fd04412f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41164/0 (socket says 172.21.15.103:41164) 2023-07-02T09:02:49.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd049621700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd04406d6e0 0x7fd044099b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57710/0 (socket says 172.21.15.103:57710) 2023-07-02T09:02:49.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd048e20700 1 -- 172.21.15.103:0/1381663186 learned_addr learned my addr 172.21.15.103:0/1381663186 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:49.904 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd049621700 1 -- 172.21.15.103:0/1381663186 >> v1:172.21.15.103:6789/0 conn(0x7fd04409a050 legacy=0x7fd04412f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:49.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd049621700 1 -- 172.21.15.103:0/1381663186 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04406dc80 con 0x7fd04406d6e0 2023-07-02T09:02:49.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd03bfff700 1 -- 172.21.15.103:0/1381663186 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131861097 0 0) 0x7fd04409cea0 con 0x7fd04409a050 2023-07-02T09:02:49.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd049621700 1 --2- 172.21.15.103:0/1381663186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd044099b10 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffdd914e57a57ca server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:49.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd03bfff700 1 -- 172.21.15.103:0/1381663186 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd040009cd0 con 0x7fd04406d6e0 2023-07-02T09:02:49.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd03bfff700 1 -- 172.21.15.103:0/1381663186 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd040009e30 con 0x7fd04406d6e0 2023-07-02T09:02:49.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.903+0000 7fd03bfff700 1 -- 172.21.15.103:0/1381663186 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd04000e360 con 0x7fd04406d6e0 2023-07-02T09:02:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.904+0000 7fd04b885700 1 -- 172.21.15.103:0/1381663186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 msgr2=0x7fd044099b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.904+0000 7fd04b885700 1 --2- 172.21.15.103:0/1381663186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd044099b10 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.904+0000 7fd04b885700 1 -- 172.21.15.103:0/1381663186 shutdown_connections 2023-07-02T09:02:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.904+0000 7fd04b885700 1 --2- 172.21.15.103:0/1381663186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd044099b10 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:49.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.904+0000 7fd04b885700 1 -- 172.21.15.103:0/1381663186 >> 172.21.15.103:0/1381663186 conn(0x7fd04409d810 msgr2=0x7fd04409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.904+0000 7fd04b885700 1 -- 172.21.15.103:0/1381663186 shutdown_connections 2023-07-02T09:02:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.904+0000 7fd04b885700 1 -- 172.21.15.103:0/1381663186 wait complete. 2023-07-02T09:02:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.904+0000 7fd04b885700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.905+0000 7fd04b885700 1 Processor -- start 2023-07-02T09:02:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.905+0000 7fd04b885700 1 -- start start 2023-07-02T09:02:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.906+0000 7fd04b885700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd0441c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:49.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.906+0000 7fd04b885700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04411f5d0 con 0x7fd04406d6e0 2023-07-02T09:02:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.906+0000 7fd049621700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd0441c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.906+0000 7fd049621700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd0441c38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57718/0 (socket says 172.21.15.103:57718) 2023-07-02T09:02:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.906+0000 7fd049621700 1 -- 172.21.15.103:0/3292755723 learned_addr learned my addr 172.21.15.103:0/3292755723 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.906+0000 7fd049621700 1 -- 172.21.15.103:0/3292755723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044124f00 con 0x7fd04406d6e0 2023-07-02T09:02:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.906+0000 7fd049621700 1 --2- 172.21.15.103:0/3292755723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd0441c38c0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.907+0000 7fd039ffb700 1 -- 172.21.15.103:0/3292755723 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd040013530 con 0x7fd04406d6e0 2023-07-02T09:02:49.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.907+0000 7fd039ffb700 1 -- 172.21.15.103:0/3292755723 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd040013690 con 0x7fd04406d6e0 2023-07-02T09:02:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.907+0000 7fd039ffb700 1 -- 172.21.15.103:0/3292755723 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd040013930 con 0x7fd04406d6e0 2023-07-02T09:02:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.907+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0441243f0 con 0x7fd04406d6e0 2023-07-02T09:02:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.907+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd044124000 con 0x7fd04406d6e0 2023-07-02T09:02:49.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.908+0000 7fd039ffb700 1 -- 172.21.15.103:0/3292755723 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd040013ae0 con 0x7fd04406d6e0 2023-07-02T09:02:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.908+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04411f420 con 0x7fd04406d6e0 2023-07-02T09:02:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.908+0000 7fd039ffb700 1 --2- 172.21.15.103:0/3292755723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0280377b0 0x7fd028039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.908+0000 7fd039ffb700 1 -- 172.21.15.103:0/3292755723 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd040018040 con 0x7fd04406d6e0 2023-07-02T09:02:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.908+0000 7fd048e20700 1 --2- 172.21.15.103:0/3292755723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0280377b0 0x7fd028039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:49.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.909+0000 7fd048e20700 1 --2- 172.21.15.103:0/3292755723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0280377b0 0x7fd028039cc0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:49.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:49.912+0000 7fd039ffb700 1 -- 172.21.15.103:0/3292755723 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd040018e00 con 0x7fd04406d6e0 2023-07-02T09:02:50.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.103+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd0441c3720 con 0x7fd04406d6e0 2023-07-02T09:02:50.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.103+0000 7fd039ffb700 1 -- 172.21.15.103:0/3292755723 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd0441c3720 con 0x7fd04406d6e0 2023-07-02T09:02:50.103 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0280377b0 msgr2=0x7fd028039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 --2- 172.21.15.103:0/3292755723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0280377b0 0x7fd028039cc0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 msgr2=0x7fd0441c38c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 --2- 172.21.15.103:0/3292755723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd0441c38c0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 shutdown_connections 2023-07-02T09:02:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 --2- 172.21.15.103:0/3292755723 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd0280377b0 0x7fd028039cc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:50.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 --2- 172.21.15.103:0/3292755723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04406d6e0 0x7fd0441c38c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:50.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 >> 172.21.15.103:0/3292755723 conn(0x7fd04409d810 msgr2=0x7fd044117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:50.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 shutdown_connections 2023-07-02T09:02:50.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:50.105+0000 7fd04b885700 1 -- 172.21.15.103:0/3292755723 wait complete. 2023-07-02T09:02:50.118 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:51.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:51.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.259+0000 7f6478cca700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:51.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.265+0000 7f6478cca700 1 Processor -- start 2023-07-02T09:02:51.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.265+0000 7f6478cca700 1 -- start start 2023-07-02T09:02:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.265+0000 7f6478cca700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6474132030 0x7f6474134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.265+0000 7f6478cca700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647406df60 con 0x7f647412f250 2023-07-02T09:02:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.265+0000 7f6478cca700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647409a710 con 0x7f6474132030 2023-07-02T09:02:51.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f6471d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6474132030 0x7f6474134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f647259c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f647412f250 0x7f647412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41170/0 (socket says 172.21.15.103:41170) 2023-07-02T09:02:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f647259c700 1 -- 172.21.15.103:0/2459786262 learned_addr learned my addr 172.21.15.103:0/2459786262 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f6471d9b700 1 -- 172.21.15.103:0/2459786262 >> v1:172.21.15.103:6789/0 conn(0x7f647412f250 legacy=0x7f647412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f6471d9b700 1 -- 172.21.15.103:0/2459786262 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647412ed20 con 0x7f6474132030 2023-07-02T09:02:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f647159a700 1 -- 172.21.15.103:0/2459786262 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894520256 0 0) 0x7f647406df60 con 0x7f647412f250 2023-07-02T09:02:51.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f6471d9b700 1 --2- 172.21.15.103:0/2459786262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6474132030 0x7f6474134450 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448b9daac20b1fb1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:51.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f647159a700 1 -- 172.21.15.103:0/2459786262 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f645c00ac60 con 0x7f6474132030 2023-07-02T09:02:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f647159a700 1 -- 172.21.15.103:0/2459786262 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f645c00adc0 con 0x7f6474132030 2023-07-02T09:02:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.266+0000 7f647159a700 1 -- 172.21.15.103:0/2459786262 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f645c00f470 con 0x7f6474132030 2023-07-02T09:02:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.267+0000 7f6478cca700 1 -- 172.21.15.103:0/2459786262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6474132030 msgr2=0x7f6474134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.267+0000 7f6478cca700 1 --2- 172.21.15.103:0/2459786262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6474132030 0x7f6474134450 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.267+0000 7f6478cca700 1 -- 172.21.15.103:0/2459786262 shutdown_connections 2023-07-02T09:02:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.267+0000 7f6478cca700 1 --2- 172.21.15.103:0/2459786262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6474132030 0x7f6474134450 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.267+0000 7f6478cca700 1 -- 172.21.15.103:0/2459786262 >> 172.21.15.103:0/2459786262 conn(0x7f647409d810 msgr2=0x7f647409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:51.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.267+0000 7f6478cca700 1 -- 172.21.15.103:0/2459786262 shutdown_connections 2023-07-02T09:02:51.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.267+0000 7f6478cca700 1 -- 172.21.15.103:0/2459786262 wait complete. 2023-07-02T09:02:51.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.267+0000 7f6478cca700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:51.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.268+0000 7f6478cca700 1 Processor -- start 2023-07-02T09:02:51.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.268+0000 7f6478cca700 1 -- start start 2023-07-02T09:02:51.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.269+0000 7f6478cca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f647412f250 0x7f647406ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.269+0000 7f6478cca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64741175d0 con 0x7f647412f250 2023-07-02T09:02:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.269+0000 7f647259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f647412f250 0x7f647406ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.269+0000 7f647259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f647412f250 0x7f647406ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57746/0 (socket says 172.21.15.103:57746) 2023-07-02T09:02:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.269+0000 7f647259c700 1 -- 172.21.15.103:0/3164044846 learned_addr learned my addr 172.21.15.103:0/3164044846 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.269+0000 7f647259c700 1 -- 172.21.15.103:0/3164044846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474119080 con 0x7f647412f250 2023-07-02T09:02:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.270+0000 7f647259c700 1 --2- 172.21.15.103:0/3164044846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f647412f250 0x7f647406ff70 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:51.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.270+0000 7f646affd700 1 -- 172.21.15.103:0/3164044846 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6464004b80 con 0x7f647412f250 2023-07-02T09:02:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.270+0000 7f646affd700 1 -- 172.21.15.103:0/3164044846 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6464004ce0 con 0x7f647412f250 2023-07-02T09:02:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.270+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64741292f0 con 0x7f647412f250 2023-07-02T09:02:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.270+0000 7f646affd700 1 -- 172.21.15.103:0/3164044846 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f646400c4f0 con 0x7f647412f250 2023-07-02T09:02:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.270+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6474117af0 con 0x7f647412f250 2023-07-02T09:02:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.271+0000 7f646affd700 1 -- 172.21.15.103:0/3164044846 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f646400c6a0 con 0x7f647412f250 2023-07-02T09:02:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.271+0000 7f646affd700 1 --2- 172.21.15.103:0/3164044846 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6454037770 0x7f6454039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.271+0000 7f646affd700 1 -- 172.21.15.103:0/3164044846 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f64640469c0 con 0x7f647412f250 2023-07-02T09:02:51.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.271+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6474061ea0 con 0x7f647412f250 2023-07-02T09:02:51.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.271+0000 7f6471d9b700 1 --2- 172.21.15.103:0/3164044846 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6454037770 0x7f6454039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:51.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.272+0000 7f6471d9b700 1 --2- 172.21.15.103:0/3164044846 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6454037770 0x7f6454039c80 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:51.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.276+0000 7f646affd700 1 -- 172.21.15.103:0/3164044846 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6464005040 con 0x7f647412f250 2023-07-02T09:02:51.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.462+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f647409cce0 con 0x7f647412f250 2023-07-02T09:02:51.463 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.463+0000 7f646affd700 1 -- 172.21.15.103:0/3164044846 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6464009c50 con 0x7f647412f250 2023-07-02T09:02:51.463 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:51.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6454037770 msgr2=0x7f6454039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:51.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 --2- 172.21.15.103:0/3164044846 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6454037770 0x7f6454039c80 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:51.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f647412f250 msgr2=0x7f647406ff70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 --2- 172.21.15.103:0/3164044846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f647412f250 0x7f647406ff70 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 shutdown_connections 2023-07-02T09:02:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 --2- 172.21.15.103:0/3164044846 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6454037770 0x7f6454039c80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 --2- 172.21.15.103:0/3164044846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f647412f250 0x7f647406ff70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 >> 172.21.15.103:0/3164044846 conn(0x7f647409d810 msgr2=0x7f647412e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 shutdown_connections 2023-07-02T09:02:51.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:51.465+0000 7f6478cca700 1 -- 172.21.15.103:0/3164044846 wait complete. 2023-07-02T09:02:51.478 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:52.479 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:52.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.618+0000 7fa5dc614700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:52.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.623+0000 7fa5dc614700 1 Processor -- start 2023-07-02T09:02:52.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.623+0000 7fa5dc614700 1 -- start start 2023-07-02T09:02:52.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5dc614700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa5d412f250 0x7fa5d412f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5dc614700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d406df60 con 0x7fa5d4098a00 2023-07-02T09:02:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5dc614700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d409a710 con 0x7fa5d412f250 2023-07-02T09:02:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5da3b0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa5d412f250 0x7fa5d412f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5d9baf700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa5d4098a00 0x7fa5d4098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41172/0 (socket says 172.21.15.103:41172) 2023-07-02T09:02:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5d9baf700 1 -- 172.21.15.103:0/1915334064 learned_addr learned my addr 172.21.15.103:0/1915334064 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5da3b0700 1 -- 172.21.15.103:0/1915334064 >> v1:172.21.15.103:6789/0 conn(0x7fa5d4098a00 legacy=0x7fa5d4098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5da3b0700 1 -- 172.21.15.103:0/1915334064 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d412ed20 con 0x7fa5d412f250 2023-07-02T09:02:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5d93ae700 1 -- 172.21.15.103:0/1915334064 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853125642 0 0) 0x7fa5d406df60 con 0x7fa5d4098a00 2023-07-02T09:02:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.624+0000 7fa5da3b0700 1 --2- 172.21.15.103:0/1915334064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d412f250 0x7fa5d412f660 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f95d5e34df20ab server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5d93ae700 1 -- 172.21.15.103:0/1915334064 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa5c4009cd0 con 0x7fa5d412f250 2023-07-02T09:02:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5d93ae700 1 -- 172.21.15.103:0/1915334064 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa5c4009e30 con 0x7fa5d412f250 2023-07-02T09:02:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5d93ae700 1 -- 172.21.15.103:0/1915334064 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa5c400e360 con 0x7fa5d412f250 2023-07-02T09:02:52.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5dc614700 1 -- 172.21.15.103:0/1915334064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d412f250 msgr2=0x7fa5d412f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:52.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5dc614700 1 --2- 172.21.15.103:0/1915334064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d412f250 0x7fa5d412f660 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:52.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5dc614700 1 -- 172.21.15.103:0/1915334064 shutdown_connections 2023-07-02T09:02:52.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5dc614700 1 --2- 172.21.15.103:0/1915334064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d412f250 0x7fa5d412f660 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:52.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5dc614700 1 -- 172.21.15.103:0/1915334064 >> 172.21.15.103:0/1915334064 conn(0x7fa5d409d810 msgr2=0x7fa5d409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:52.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5dc614700 1 -- 172.21.15.103:0/1915334064 shutdown_connections 2023-07-02T09:02:52.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.625+0000 7fa5dc614700 1 -- 172.21.15.103:0/1915334064 wait complete. 2023-07-02T09:02:52.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.626+0000 7fa5dc614700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.627+0000 7fa5dc614700 1 Processor -- start 2023-07-02T09:02:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.627+0000 7fa5dc614700 1 -- start start 2023-07-02T09:02:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5dc614700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d4098a00 0x7fa5d4117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:52.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5dc614700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d411a390 con 0x7fa5d4098a00 2023-07-02T09:02:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5da3b0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d4098a00 0x7fa5d4117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5da3b0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d4098a00 0x7fa5d4117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57774/0 (socket says 172.21.15.103:57774) 2023-07-02T09:02:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5da3b0700 1 -- 172.21.15.103:0/3869979196 learned_addr learned my addr 172.21.15.103:0/3869979196 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5da3b0700 1 -- 172.21.15.103:0/3869979196 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4099b60 con 0x7fa5d4098a00 2023-07-02T09:02:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5da3b0700 1 --2- 172.21.15.103:0/3869979196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d4098a00 0x7fa5d4117360 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5caffd700 1 -- 172.21.15.103:0/3869979196 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa5c4014540 con 0x7fa5d4098a00 2023-07-02T09:02:52.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5caffd700 1 -- 172.21.15.103:0/3869979196 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa5c40146a0 con 0x7fa5d4098a00 2023-07-02T09:02:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5caffd700 1 -- 172.21.15.103:0/3869979196 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa5c4014940 con 0x7fa5d4098a00 2023-07-02T09:02:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.628+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d41196f0 con 0x7fa5d4098a00 2023-07-02T09:02:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.629+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d4119db0 con 0x7fa5d4098a00 2023-07-02T09:02:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.629+0000 7fa5caffd700 1 -- 172.21.15.103:0/3869979196 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa5c4014af0 con 0x7fa5d4098a00 2023-07-02T09:02:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.629+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d4061ea0 con 0x7fa5d4098a00 2023-07-02T09:02:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.629+0000 7fa5caffd700 1 --2- 172.21.15.103:0/3869979196 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5b80377b0 0x7fa5b8039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.630+0000 7fa5caffd700 1 -- 172.21.15.103:0/3869979196 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa5c4048200 con 0x7fa5d4098a00 2023-07-02T09:02:52.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.630+0000 7fa5d9baf700 1 --2- 172.21.15.103:0/3869979196 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5b80377b0 0x7fa5b8039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:52.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.630+0000 7fa5d9baf700 1 --2- 172.21.15.103:0/3869979196 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5b80377b0 0x7fa5b8039cc0 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:52.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.634+0000 7fa5caffd700 1 -- 172.21.15.103:0/3869979196 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa5c400fb00 con 0x7fa5d4098a00 2023-07-02T09:02:52.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.825+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa5d4099b60 con 0x7fa5d4098a00 2023-07-02T09:02:52.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.826+0000 7fa5caffd700 1 -- 172.21.15.103:0/3869979196 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa5d4099b60 con 0x7fa5d4098a00 2023-07-02T09:02:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:52.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5b80377b0 msgr2=0x7fa5b8039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:52.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 --2- 172.21.15.103:0/3869979196 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5b80377b0 0x7fa5b8039cc0 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:52.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d4098a00 msgr2=0x7fa5d4117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:52.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 --2- 172.21.15.103:0/3869979196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d4098a00 0x7fa5d4117360 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:52.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 shutdown_connections 2023-07-02T09:02:52.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 --2- 172.21.15.103:0/3869979196 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa5b80377b0 0x7fa5b8039cc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:52.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 --2- 172.21.15.103:0/3869979196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5d4098a00 0x7fa5d4117360 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:52.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 >> 172.21.15.103:0/3869979196 conn(0x7fa5d409d810 msgr2=0x7fa5d40a0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:52.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 shutdown_connections 2023-07-02T09:02:52.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:52.828+0000 7fa5dc614700 1 -- 172.21.15.103:0/3869979196 wait complete. 2023-07-02T09:02:52.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:53.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~187s 2023-07-02T09:02:53.842 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:53.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.982+0000 7f7a3bce4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:53.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.987+0000 7f7a3bce4700 1 Processor -- start 2023-07-02T09:02:53.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.987+0000 7f7a3bce4700 1 -- start start 2023-07-02T09:02:53.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a3bce4700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7a34098a00 0x7f7a34098e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a3bce4700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3406f8b0 con 0x7f7a34099350 2023-07-02T09:02:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a3bce4700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a34072510 con 0x7f7a34098a00 2023-07-02T09:02:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a39a80700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7a34098a00 0x7f7a34098e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a3927f700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7a34099350 0x7f7a3412d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42598/0 (socket says 172.21.15.103:42598) 2023-07-02T09:02:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a3927f700 1 -- 172.21.15.103:0/1667480337 learned_addr learned my addr 172.21.15.103:0/1667480337 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a39a80700 1 -- 172.21.15.103:0/1667480337 >> v1:172.21.15.103:6789/0 conn(0x7f7a34099350 legacy=0x7f7a3412d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a39a80700 1 -- 172.21.15.103:0/1667480337 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3409aec0 con 0x7f7a34098a00 2023-07-02T09:02:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a39a80700 1 --2- 172.21.15.103:0/1667480337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a34098e10 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429a9970dd3636ef server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a38a7e700 1 -- 172.21.15.103:0/1667480337 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a30009cd0 con 0x7f7a34098a00 2023-07-02T09:02:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a38a7e700 1 -- 172.21.15.103:0/1667480337 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7a30009e30 con 0x7f7a34098a00 2023-07-02T09:02:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.988+0000 7f7a38a7e700 1 -- 172.21.15.103:0/1667480337 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a3000e360 con 0x7f7a34098a00 2023-07-02T09:02:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.989+0000 7f7a3bce4700 1 -- 172.21.15.103:0/1667480337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 msgr2=0x7f7a34098e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.989+0000 7f7a3bce4700 1 --2- 172.21.15.103:0/1667480337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a34098e10 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.989+0000 7f7a3bce4700 1 -- 172.21.15.103:0/1667480337 shutdown_connections 2023-07-02T09:02:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.989+0000 7f7a3bce4700 1 --2- 172.21.15.103:0/1667480337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a34098e10 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.989+0000 7f7a3bce4700 1 -- 172.21.15.103:0/1667480337 >> 172.21.15.103:0/1667480337 conn(0x7f7a3409d810 msgr2=0x7f7a3409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.989+0000 7f7a3bce4700 1 -- 172.21.15.103:0/1667480337 shutdown_connections 2023-07-02T09:02:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.989+0000 7f7a3bce4700 1 -- 172.21.15.103:0/1667480337 wait complete. 2023-07-02T09:02:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.989+0000 7f7a3bce4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.990+0000 7f7a3bce4700 1 Processor -- start 2023-07-02T09:02:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.990+0000 7f7a3bce4700 1 -- start start 2023-07-02T09:02:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.991+0000 7f7a3bce4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a341c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.991+0000 7f7a3bce4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3411cc50 con 0x7f7a34098a00 2023-07-02T09:02:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.991+0000 7f7a39a80700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a341c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.991+0000 7f7a39a80700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a341c4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56200/0 (socket says 172.21.15.103:56200) 2023-07-02T09:02:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.991+0000 7f7a39a80700 1 -- 172.21.15.103:0/3729565705 learned_addr learned my addr 172.21.15.103:0/3729565705 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.991+0000 7f7a39a80700 1 -- 172.21.15.103:0/3729565705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3411ca20 con 0x7f7a34098a00 2023-07-02T09:02:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.992+0000 7f7a39a80700 1 --2- 172.21.15.103:0/3729565705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a341c4ea0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.992+0000 7f7a267fc700 1 -- 172.21.15.103:0/3729565705 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a30013740 con 0x7f7a34098a00 2023-07-02T09:02:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.992+0000 7f7a267fc700 1 -- 172.21.15.103:0/3729565705 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7a300138a0 con 0x7f7a34098a00 2023-07-02T09:02:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.992+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a341b9bd0 con 0x7f7a34098a00 2023-07-02T09:02:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.992+0000 7f7a267fc700 1 -- 172.21.15.103:0/3729565705 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a30013b40 con 0x7f7a34098a00 2023-07-02T09:02:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.992+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3411d3a0 con 0x7f7a34098a00 2023-07-02T09:02:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.993+0000 7f7a267fc700 1 -- 172.21.15.103:0/3729565705 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7a30013cf0 con 0x7f7a34098a00 2023-07-02T09:02:53.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.993+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3411d920 con 0x7f7a34098a00 2023-07-02T09:02:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.993+0000 7f7a267fc700 1 --2- 172.21.15.103:0/3729565705 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a180377c0 0x7f7a18039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.993+0000 7f7a267fc700 1 -- 172.21.15.103:0/3729565705 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7a30018040 con 0x7f7a34098a00 2023-07-02T09:02:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.993+0000 7f7a3927f700 1 --2- 172.21.15.103:0/3729565705 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a180377c0 0x7f7a18039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.994+0000 7f7a3927f700 1 --2- 172.21.15.103:0/3729565705 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a180377c0 0x7f7a18039cd0 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:53.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:53.998+0000 7f7a267fc700 1 -- 172.21.15.103:0/3729565705 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7a300189b0 con 0x7f7a34098a00 2023-07-02T09:02:54.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.187+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7a34061ea0 con 0x7f7a34098a00 2023-07-02T09:02:54.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.187+0000 7f7a267fc700 1 -- 172.21.15.103:0/3729565705 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7a3000f2a0 con 0x7f7a34098a00 2023-07-02T09:02:54.188 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:54.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a180377c0 msgr2=0x7f7a18039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:54.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 --2- 172.21.15.103:0/3729565705 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a180377c0 0x7f7a18039cd0 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:54.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 msgr2=0x7f7a341c4ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:54.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 --2- 172.21.15.103:0/3729565705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a341c4ea0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:54.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 shutdown_connections 2023-07-02T09:02:54.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 --2- 172.21.15.103:0/3729565705 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a180377c0 0x7f7a18039cd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:54.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 --2- 172.21.15.103:0/3729565705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a34098a00 0x7f7a341c4ea0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:54.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 >> 172.21.15.103:0/3729565705 conn(0x7f7a3409d810 msgr2=0x7f7a341b8790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:54.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 shutdown_connections 2023-07-02T09:02:54.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:54.189+0000 7f7a3bce4700 1 -- 172.21.15.103:0/3729565705 wait complete. 2023-07-02T09:02:54.202 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:55.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:55.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.344+0000 7f4c03be2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:55.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.349+0000 7f4c03be2700 1 Processor -- start 2023-07-02T09:02:55.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.349+0000 7f4c03be2700 1 -- start start 2023-07-02T09:02:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c03be2700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4bfc132030 0x7f4bfc134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c03be2700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bfc06df60 con 0x7f4bfc12f250 2023-07-02T09:02:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c03be2700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bfc09a710 con 0x7f4bfc132030 2023-07-02T09:02:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c0117d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4bfc132030 0x7f4bfc134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c0197e700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4bfc12f250 0x7f4bfc12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42608/0 (socket says 172.21.15.103:42608) 2023-07-02T09:02:55.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c0117d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4bfc132030 0x7f4bfc134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56210/0 (socket says 172.21.15.103:56210) 2023-07-02T09:02:55.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c0197e700 1 -- 172.21.15.103:0/489915104 learned_addr learned my addr 172.21.15.103:0/489915104 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:55.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c0117d700 1 -- 172.21.15.103:0/489915104 >> v1:172.21.15.103:6789/0 conn(0x7f4bfc12f250 legacy=0x7f4bfc12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:55.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c0117d700 1 -- 172.21.15.103:0/489915104 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc12ed20 con 0x7f4bfc132030 2023-07-02T09:02:55.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.350+0000 7f4c0097c700 1 -- 172.21.15.103:0/489915104 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391801673 0 0) 0x7f4bfc06df60 con 0x7f4bfc12f250 2023-07-02T09:02:55.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c0117d700 1 --2- 172.21.15.103:0/489915104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc132030 0x7f4bfc134450 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526d7b7dd9d88792 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:55.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c0097c700 1 -- 172.21.15.103:0/489915104 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4bf800d450 con 0x7f4bfc132030 2023-07-02T09:02:55.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c0097c700 1 -- 172.21.15.103:0/489915104 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4bf800d5b0 con 0x7f4bfc132030 2023-07-02T09:02:55.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c0097c700 1 -- 172.21.15.103:0/489915104 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4bf800d850 con 0x7f4bfc132030 2023-07-02T09:02:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c03be2700 1 -- 172.21.15.103:0/489915104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc132030 msgr2=0x7f4bfc134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c03be2700 1 --2- 172.21.15.103:0/489915104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc132030 0x7f4bfc134450 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:55.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c03be2700 1 -- 172.21.15.103:0/489915104 shutdown_connections 2023-07-02T09:02:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c03be2700 1 --2- 172.21.15.103:0/489915104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc132030 0x7f4bfc134450 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c03be2700 1 -- 172.21.15.103:0/489915104 >> 172.21.15.103:0/489915104 conn(0x7f4bfc09d810 msgr2=0x7f4bfc09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.351+0000 7f4c03be2700 1 -- 172.21.15.103:0/489915104 shutdown_connections 2023-07-02T09:02:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.352+0000 7f4c03be2700 1 -- 172.21.15.103:0/489915104 wait complete. 2023-07-02T09:02:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.352+0000 7f4c03be2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.353+0000 7f4c03be2700 1 Processor -- start 2023-07-02T09:02:55.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.353+0000 7f4c03be2700 1 -- start start 2023-07-02T09:02:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.353+0000 7f4c03be2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc12f250 0x7f4bfc06ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.353+0000 7f4c03be2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc1175d0 con 0x7f4bfc12f250 2023-07-02T09:02:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.353+0000 7f4c0197e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc12f250 0x7f4bfc06ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.353+0000 7f4c0197e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc12f250 0x7f4bfc06ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56218/0 (socket says 172.21.15.103:56218) 2023-07-02T09:02:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.353+0000 7f4c0197e700 1 -- 172.21.15.103:0/3162172427 learned_addr learned my addr 172.21.15.103:0/3162172427 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.354+0000 7f4c0197e700 1 -- 172.21.15.103:0/3162172427 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc119080 con 0x7f4bfc12f250 2023-07-02T09:02:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.354+0000 7f4c0197e700 1 --2- 172.21.15.103:0/3162172427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc12f250 0x7f4bfc06ff70 crc :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:55.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.354+0000 7f4bf27fc700 1 -- 172.21.15.103:0/3162172427 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4bec004b80 con 0x7f4bfc12f250 2023-07-02T09:02:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.354+0000 7f4bf27fc700 1 -- 172.21.15.103:0/3162172427 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4bec004ce0 con 0x7f4bfc12f250 2023-07-02T09:02:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.354+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bfc1292f0 con 0x7f4bfc12f250 2023-07-02T09:02:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.354+0000 7f4bf27fc700 1 -- 172.21.15.103:0/3162172427 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4bec00c4f0 con 0x7f4bfc12f250 2023-07-02T09:02:55.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.354+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bfc117af0 con 0x7f4bfc12f250 2023-07-02T09:02:55.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.355+0000 7f4bf27fc700 1 -- 172.21.15.103:0/3162172427 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4bec00c650 con 0x7f4bfc12f250 2023-07-02T09:02:55.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.355+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bfc061ea0 con 0x7f4bfc12f250 2023-07-02T09:02:55.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.355+0000 7f4bf27fc700 1 --2- 172.21.15.103:0/3162172427 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4be00377c0 0x7f4be0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:55.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.355+0000 7f4bf27fc700 1 -- 172.21.15.103:0/3162172427 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4bec046970 con 0x7f4bfc12f250 2023-07-02T09:02:55.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.355+0000 7f4c0117d700 1 --2- 172.21.15.103:0/3162172427 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4be00377c0 0x7f4be0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:55.359 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.356+0000 7f4c0117d700 1 --2- 172.21.15.103:0/3162172427 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4be00377c0 0x7f4be0039cd0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:55.360 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.360+0000 7f4bf27fc700 1 -- 172.21.15.103:0/3162172427 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f4bec009c80 con 0x7f4bfc12f250 2023-07-02T09:02:55.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.549+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4bfc09cce0 con 0x7f4bfc12f250 2023-07-02T09:02:55.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.549+0000 7f4bf27fc700 1 -- 172.21.15.103:0/3162172427 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4bec0098a0 con 0x7f4bfc12f250 2023-07-02T09:02:55.550 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4be00377c0 msgr2=0x7f4be0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 --2- 172.21.15.103:0/3162172427 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4be00377c0 0x7f4be0039cd0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc12f250 msgr2=0x7f4bfc06ff70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 --2- 172.21.15.103:0/3162172427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc12f250 0x7f4bfc06ff70 crc :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 shutdown_connections 2023-07-02T09:02:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 --2- 172.21.15.103:0/3162172427 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4be00377c0 0x7f4be0039cd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:55.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 --2- 172.21.15.103:0/3162172427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bfc12f250 0x7f4bfc06ff70 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:55.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 >> 172.21.15.103:0/3162172427 conn(0x7f4bfc09d810 msgr2=0x7f4bfc12e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:55.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 shutdown_connections 2023-07-02T09:02:55.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:55.551+0000 7f4c03be2700 1 -- 172.21.15.103:0/3162172427 wait complete. 2023-07-02T09:02:55.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:56.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:56.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.704+0000 7f08b7e50700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:56.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.709+0000 7f08b7e50700 1 Processor -- start 2023-07-02T09:02:56.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.710+0000 7f08b7e50700 1 -- start start 2023-07-02T09:02:56.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.710+0000 7f08b7e50700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f08b0133360 0x7f08b009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:56.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.710+0000 7f08b7e50700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b009e260 con 0x7f08b0132f50 2023-07-02T09:02:56.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.710+0000 7f08b7e50700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b00a1640 con 0x7f08b0133360 2023-07-02T09:02:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.710+0000 7f08b53eb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f08b0133360 0x7f08b009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b5bec700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f08b0132f50 0x7f08b009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42610/0 (socket says 172.21.15.103:42610) 2023-07-02T09:02:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b5bec700 1 -- 172.21.15.103:0/997005674 learned_addr learned my addr 172.21.15.103:0/997005674 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b53eb700 1 -- 172.21.15.103:0/997005674 >> v1:172.21.15.103:6789/0 conn(0x7f08b0132f50 legacy=0x7f08b009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b53eb700 1 -- 172.21.15.103:0/997005674 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b0132a20 con 0x7f08b0133360 2023-07-02T09:02:56.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b4bea700 1 -- 172.21.15.103:0/997005674 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274002091 0 0) 0x7f08b009e260 con 0x7f08b0132f50 2023-07-02T09:02:56.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b53eb700 1 --2- 172.21.15.103:0/997005674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0133360 0x7f08b009cc90 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b591f0082a5331 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:56.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b4bea700 1 -- 172.21.15.103:0/997005674 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f08ac009cd0 con 0x7f08b0133360 2023-07-02T09:02:56.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b4bea700 1 -- 172.21.15.103:0/997005674 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f08ac009e30 con 0x7f08b0133360 2023-07-02T09:02:56.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.711+0000 7f08b4bea700 1 -- 172.21.15.103:0/997005674 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f08ac00e360 con 0x7f08b0133360 2023-07-02T09:02:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.712+0000 7f08b7e50700 1 -- 172.21.15.103:0/997005674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0133360 msgr2=0x7f08b009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.712+0000 7f08b7e50700 1 --2- 172.21.15.103:0/997005674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0133360 0x7f08b009cc90 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.712+0000 7f08b7e50700 1 -- 172.21.15.103:0/997005674 shutdown_connections 2023-07-02T09:02:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.712+0000 7f08b7e50700 1 --2- 172.21.15.103:0/997005674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0133360 0x7f08b009cc90 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.712+0000 7f08b7e50700 1 -- 172.21.15.103:0/997005674 >> 172.21.15.103:0/997005674 conn(0x7f08b00a2080 msgr2=0x7f08b00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.712+0000 7f08b7e50700 1 -- 172.21.15.103:0/997005674 shutdown_connections 2023-07-02T09:02:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.712+0000 7f08b7e50700 1 -- 172.21.15.103:0/997005674 wait complete. 2023-07-02T09:02:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.712+0000 7f08b7e50700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.713+0000 7f08b7e50700 1 Processor -- start 2023-07-02T09:02:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.713+0000 7f08b7e50700 1 -- start start 2023-07-02T09:02:56.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b7e50700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0132f50 0x7f08b012c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b7e50700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b006d2f0 con 0x7f08b0132f50 2023-07-02T09:02:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b5bec700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0132f50 0x7f08b012c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b5bec700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0132f50 0x7f08b012c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56238/0 (socket says 172.21.15.103:56238) 2023-07-02T09:02:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b5bec700 1 -- 172.21.15.103:0/1669541714 learned_addr learned my addr 172.21.15.103:0/1669541714 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b5bec700 1 -- 172.21.15.103:0/1669541714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b006dd60 con 0x7f08b0132f50 2023-07-02T09:02:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b5bec700 1 --2- 172.21.15.103:0/1669541714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0132f50 0x7f08b012c1a0 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08a67fc700 1 -- 172.21.15.103:0/1669541714 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f08a0004b80 con 0x7f08b0132f50 2023-07-02T09:02:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08a67fc700 1 -- 172.21.15.103:0/1669541714 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f08a0004ce0 con 0x7f08b0132f50 2023-07-02T09:02:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b012bc50 con 0x7f08b0132f50 2023-07-02T09:02:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08a67fc700 1 -- 172.21.15.103:0/1669541714 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f08a000c4f0 con 0x7f08b0132f50 2023-07-02T09:02:56.719 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.714+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b006f850 con 0x7f08b0132f50 2023-07-02T09:02:56.719 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.715+0000 7f08a67fc700 1 -- 172.21.15.103:0/1669541714 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f08a000c650 con 0x7f08b0132f50 2023-07-02T09:02:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.715+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b006dbb0 con 0x7f08b0132f50 2023-07-02T09:02:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.715+0000 7f08a67fc700 1 --2- 172.21.15.103:0/1669541714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f08940377c0 0x7f0894039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.715+0000 7f08a67fc700 1 -- 172.21.15.103:0/1669541714 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f08a00469f0 con 0x7f08b0132f50 2023-07-02T09:02:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.716+0000 7f08b53eb700 1 --2- 172.21.15.103:0/1669541714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f08940377c0 0x7f0894039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.716+0000 7f08b53eb700 1 --2- 172.21.15.103:0/1669541714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f08940377c0 0x7f0894039cd0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:56.721 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.720+0000 7f08a67fc700 1 -- 172.21.15.103:0/1669541714 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f08a0009c80 con 0x7f08b0132f50 2023-07-02T09:02:56.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.909+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f08b0061ea0 con 0x7f08b0132f50 2023-07-02T09:02:56.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.909+0000 7f08a67fc700 1 -- 172.21.15.103:0/1669541714 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f08a00098a0 con 0x7f08b0132f50 2023-07-02T09:02:56.910 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:56.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f08940377c0 msgr2=0x7f0894039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:56.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 --2- 172.21.15.103:0/1669541714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f08940377c0 0x7f0894039cd0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0132f50 msgr2=0x7f08b012c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 --2- 172.21.15.103:0/1669541714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0132f50 0x7f08b012c1a0 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 shutdown_connections 2023-07-02T09:02:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 --2- 172.21.15.103:0/1669541714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f08940377c0 0x7f0894039cd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 --2- 172.21.15.103:0/1669541714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08b0132f50 0x7f08b012c1a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 >> 172.21.15.103:0/1669541714 conn(0x7f08b00a2080 msgr2=0x7f08b0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 shutdown_connections 2023-07-02T09:02:56.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:56.911+0000 7f08b7e50700 1 -- 172.21.15.103:0/1669541714 wait complete. 2023-07-02T09:02:56.924 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:57.925 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:58.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.064+0000 7f14d6664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:58.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.069+0000 7f14d6664700 1 Processor -- start 2023-07-02T09:02:58.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.069+0000 7f14d6664700 1 -- start start 2023-07-02T09:02:58.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.070+0000 7f14d6664700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14d0132f50 0x7f14d0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:58.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.070+0000 7f14d6664700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d009e260 con 0x7f14d009c580 2023-07-02T09:02:58.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.070+0000 7f14d6664700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d00a1640 con 0x7f14d0132f50 2023-07-02T09:02:58.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.070+0000 7f14cffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14d0132f50 0x7f14d0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:58.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.070+0000 7f14cffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14d0132f50 0x7f14d0133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56240/0 (socket says 172.21.15.103:56240) 2023-07-02T09:02:58.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.070+0000 7f14cffff700 1 -- 172.21.15.103:0/2792018790 learned_addr learned my addr 172.21.15.103:0/2792018790 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:58.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.070+0000 7f14cf7fe700 1 --1- 172.21.15.103:0/2792018790 >> v1:172.21.15.103:6789/0 conn(0x7f14d009c580 0x7f14d009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42624/0 (socket says 172.21.15.103:42624) 2023-07-02T09:02:58.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.071+0000 7f14cffff700 1 -- 172.21.15.103:0/2792018790 >> v1:172.21.15.103:6789/0 conn(0x7f14d009c580 legacy=0x7f14d009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.071+0000 7f14cffff700 1 -- 172.21.15.103:0/2792018790 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d0132a20 con 0x7f14d0132f50 2023-07-02T09:02:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.071+0000 7f14ceffd700 1 -- 172.21.15.103:0/2792018790 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577906974 0 0) 0x7f14d009e260 con 0x7f14d009c580 2023-07-02T09:02:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.071+0000 7f14cffff700 1 --2- 172.21.15.103:0/2792018790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d0132f50 0x7f14d0133360 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16deaeab8fc0992f server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.071+0000 7f14ceffd700 1 -- 172.21.15.103:0/2792018790 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f14c0009d90 con 0x7f14d0132f50 2023-07-02T09:02:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.071+0000 7f14ceffd700 1 -- 172.21.15.103:0/2792018790 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f14c000e230 con 0x7f14d0132f50 2023-07-02T09:02:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.071+0000 7f14ceffd700 1 -- 172.21.15.103:0/2792018790 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f14c000e4d0 con 0x7f14d0132f50 2023-07-02T09:02:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.072+0000 7f14d6664700 1 -- 172.21.15.103:0/2792018790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d0132f50 msgr2=0x7f14d0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.072+0000 7f14d6664700 1 --2- 172.21.15.103:0/2792018790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d0132f50 0x7f14d0133360 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.072+0000 7f14d6664700 1 -- 172.21.15.103:0/2792018790 shutdown_connections 2023-07-02T09:02:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.072+0000 7f14d6664700 1 --2- 172.21.15.103:0/2792018790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d0132f50 0x7f14d0133360 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.072+0000 7f14d6664700 1 -- 172.21.15.103:0/2792018790 >> 172.21.15.103:0/2792018790 conn(0x7f14d00a2080 msgr2=0x7f14d00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.072+0000 7f14d6664700 1 -- 172.21.15.103:0/2792018790 shutdown_connections 2023-07-02T09:02:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.072+0000 7f14d6664700 1 -- 172.21.15.103:0/2792018790 wait complete. 2023-07-02T09:02:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.072+0000 7f14d6664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.073+0000 7f14d6664700 1 Processor -- start 2023-07-02T09:02:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.073+0000 7f14d6664700 1 -- start start 2023-07-02T09:02:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14d6664700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d009c580 0x7f14d01baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14d6664700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d01ba610 con 0x7f14d009c580 2023-07-02T09:02:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14cffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d009c580 0x7f14d01baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14cffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d009c580 0x7f14d01baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56242/0 (socket says 172.21.15.103:56242) 2023-07-02T09:02:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14cffff700 1 -- 172.21.15.103:0/269983468 learned_addr learned my addr 172.21.15.103:0/269983468 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14cffff700 1 -- 172.21.15.103:0/269983468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d006dfc0 con 0x7f14d009c580 2023-07-02T09:02:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14cffff700 1 --2- 172.21.15.103:0/269983468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d009c580 0x7f14d01baa60 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14ccff9700 1 -- 172.21.15.103:0/269983468 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f14c0014410 con 0x7f14d009c580 2023-07-02T09:02:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14ccff9700 1 -- 172.21.15.103:0/269983468 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f14c0014570 con 0x7f14d009c580 2023-07-02T09:02:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14ccff9700 1 -- 172.21.15.103:0/269983468 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f14c0014810 con 0x7f14d009c580 2023-07-02T09:02:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d01ba270 con 0x7f14d009c580 2023-07-02T09:02:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.074+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d006dba0 con 0x7f14d009c580 2023-07-02T09:02:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.075+0000 7f14ccff9700 1 -- 172.21.15.103:0/269983468 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f14c00149c0 con 0x7f14d009c580 2023-07-02T09:02:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.075+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d01ba460 con 0x7f14d009c580 2023-07-02T09:02:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.076+0000 7f14ccff9700 1 --2- 172.21.15.103:0/269983468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377c0 0x7f14b0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.076+0000 7f14ccff9700 1 -- 172.21.15.103:0/269983468 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f14c00482f0 con 0x7f14d009c580 2023-07-02T09:02:58.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.076+0000 7f14cf7fe700 1 --2- 172.21.15.103:0/269983468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377c0 0x7f14b0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:58.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.076+0000 7f14cf7fe700 1 --2- 172.21.15.103:0/269983468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377c0 0x7f14b0039cd0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:58.080 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.080+0000 7f14ccff9700 1 -- 172.21.15.103:0/269983468 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f14c000f8f0 con 0x7f14d009c580 2023-07-02T09:02:58.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.271+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f14d0061ea0 con 0x7f14d009c580 2023-07-02T09:02:58.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.271+0000 7f14ccff9700 1 -- 172.21.15.103:0/269983468 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f14c000f710 con 0x7f14d009c580 2023-07-02T09:02:58.272 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.273+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377c0 msgr2=0x7f14b0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.273+0000 7f14d6664700 1 --2- 172.21.15.103:0/269983468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377c0 0x7f14b0039cd0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.273+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d009c580 msgr2=0x7f14d01baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.273+0000 7f14d6664700 1 --2- 172.21.15.103:0/269983468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d009c580 0x7f14d01baa60 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.273+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 shutdown_connections 2023-07-02T09:02:58.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.273+0000 7f14d6664700 1 --2- 172.21.15.103:0/269983468 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377c0 0x7f14b0039cd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:58.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.273+0000 7f14d6664700 1 --2- 172.21.15.103:0/269983468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d009c580 0x7f14d01baa60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:58.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.273+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 >> 172.21.15.103:0/269983468 conn(0x7f14d00a2080 msgr2=0x7f14d0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:58.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.274+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 shutdown_connections 2023-07-02T09:02:58.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:58.274+0000 7f14d6664700 1 -- 172.21.15.103:0/269983468 wait complete. 2023-07-02T09:02:58.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:02:58.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~193s 2023-07-02T09:02:59.288 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:02:59.432 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.428+0000 7f4728cd4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:59.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.432+0000 7f4728cd4700 1 Processor -- start 2023-07-02T09:02:59.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.433+0000 7f4728cd4700 1 -- start start 2023-07-02T09:02:59.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.433+0000 7f4728cd4700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f472412f250 0x7f472412f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:59.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.433+0000 7f4728cd4700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472406df60 con 0x7f4724098a00 2023-07-02T09:02:59.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.433+0000 7f4728cd4700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472409a710 con 0x7f472412f250 2023-07-02T09:02:59.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f472259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f472412f250 0x7f472412f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:59.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f4721d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4724098a00 0x7f4724098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42626/0 (socket says 172.21.15.103:42626) 2023-07-02T09:02:59.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f4721d9b700 1 -- 172.21.15.103:0/4223697610 learned_addr learned my addr 172.21.15.103:0/4223697610 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:02:59.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f472259c700 1 -- 172.21.15.103:0/4223697610 >> v1:172.21.15.103:6789/0 conn(0x7f4724098a00 legacy=0x7f4724098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:59.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f472259c700 1 -- 172.21.15.103:0/4223697610 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472412ed20 con 0x7f472412f250 2023-07-02T09:02:59.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f472259c700 1 --2- 172.21.15.103:0/4223697610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472412f250 0x7f472412f660 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef0f3bcde61e5ad server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:59.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f472159a700 1 -- 172.21.15.103:0/4223697610 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f471800ac60 con 0x7f472412f250 2023-07-02T09:02:59.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f472159a700 1 -- 172.21.15.103:0/4223697610 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f471800adc0 con 0x7f472412f250 2023-07-02T09:02:59.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.434+0000 7f472159a700 1 -- 172.21.15.103:0/4223697610 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f471800f470 con 0x7f472412f250 2023-07-02T09:02:59.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.435+0000 7f4728cd4700 1 -- 172.21.15.103:0/4223697610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472412f250 msgr2=0x7f472412f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:59.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.435+0000 7f4728cd4700 1 --2- 172.21.15.103:0/4223697610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472412f250 0x7f472412f660 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:59.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.435+0000 7f4728cd4700 1 -- 172.21.15.103:0/4223697610 shutdown_connections 2023-07-02T09:02:59.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.435+0000 7f4728cd4700 1 --2- 172.21.15.103:0/4223697610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f472412f250 0x7f472412f660 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:59.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.435+0000 7f4728cd4700 1 -- 172.21.15.103:0/4223697610 >> 172.21.15.103:0/4223697610 conn(0x7f472409d810 msgr2=0x7f472409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:59.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.435+0000 7f4728cd4700 1 -- 172.21.15.103:0/4223697610 shutdown_connections 2023-07-02T09:02:59.438 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.435+0000 7f4728cd4700 1 -- 172.21.15.103:0/4223697610 wait complete. 2023-07-02T09:02:59.438 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.435+0000 7f4728cd4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:02:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.436+0000 7f4728cd4700 1 Processor -- start 2023-07-02T09:02:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.436+0000 7f4728cd4700 1 -- start start 2023-07-02T09:02:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.437+0000 7f4728cd4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4724098a00 0x7f4724117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.437+0000 7f4728cd4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472411a9f0 con 0x7f4724098a00 2023-07-02T09:02:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.437+0000 7f472259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4724098a00 0x7f4724117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.437+0000 7f472259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4724098a00 0x7f4724117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56264/0 (socket says 172.21.15.103:56264) 2023-07-02T09:02:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.437+0000 7f472259c700 1 -- 172.21.15.103:0/936346919 learned_addr learned my addr 172.21.15.103:0/936346919 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:02:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.437+0000 7f472259c700 1 -- 172.21.15.103:0/936346919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724118f50 con 0x7f4724098a00 2023-07-02T09:02:59.441 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.437+0000 7f472259c700 1 --2- 172.21.15.103:0/936346919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4724098a00 0x7f4724117360 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:59.441 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.438+0000 7f4712ffd700 1 -- 172.21.15.103:0/936346919 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4718018040 con 0x7f4724098a00 2023-07-02T09:02:59.441 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.438+0000 7f4712ffd700 1 -- 172.21.15.103:0/936346919 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4718013e80 con 0x7f4724098a00 2023-07-02T09:02:59.441 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.438+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472411a650 con 0x7f4724098a00 2023-07-02T09:02:59.441 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.438+0000 7f4712ffd700 1 -- 172.21.15.103:0/936346919 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f471800f750 con 0x7f4724098a00 2023-07-02T09:02:59.441 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.438+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472411afe0 con 0x7f4724098a00 2023-07-02T09:02:59.442 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.439+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472411a840 con 0x7f4724098a00 2023-07-02T09:02:59.442 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.439+0000 7f4712ffd700 1 -- 172.21.15.103:0/936346919 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f471801b040 con 0x7f4724098a00 2023-07-02T09:02:59.442 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.439+0000 7f4712ffd700 1 --2- 172.21.15.103:0/936346919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f47040377c0 0x7f4704039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:02:59.442 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.439+0000 7f4712ffd700 1 -- 172.21.15.103:0/936346919 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4718049cc0 con 0x7f4724098a00 2023-07-02T09:02:59.442 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.439+0000 7f4721d9b700 1 --2- 172.21.15.103:0/936346919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f47040377c0 0x7f4704039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:02:59.442 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.440+0000 7f4721d9b700 1 --2- 172.21.15.103:0/936346919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f47040377c0 0x7f4704039cd0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:02:59.443 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.443+0000 7f4712ffd700 1 -- 172.21.15.103:0/936346919 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f471800de00 con 0x7f4724098a00 2023-07-02T09:02:59.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.635+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f47241198a0 con 0x7f4724098a00 2023-07-02T09:02:59.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.635+0000 7f4712ffd700 1 -- 172.21.15.103:0/936346919 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f471801a280 con 0x7f4724098a00 2023-07-02T09:02:59.636 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:02:59.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f47040377c0 msgr2=0x7f4704039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:59.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 --2- 172.21.15.103:0/936346919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f47040377c0 0x7f4704039cd0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:59.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4724098a00 msgr2=0x7f4724117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:02:59.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 --2- 172.21.15.103:0/936346919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4724098a00 0x7f4724117360 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:59.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 shutdown_connections 2023-07-02T09:02:59.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 --2- 172.21.15.103:0/936346919 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f47040377c0 0x7f4704039cd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:59.638 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 --2- 172.21.15.103:0/936346919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4724098a00 0x7f4724117360 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:02:59.639 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 >> 172.21.15.103:0/936346919 conn(0x7f472409d810 msgr2=0x7f472409f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:02:59.639 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 shutdown_connections 2023-07-02T09:02:59.639 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:02:59.637+0000 7f4728cd4700 1 -- 172.21.15.103:0/936346919 wait complete. 2023-07-02T09:02:59.650 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:00.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:00.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.791+0000 7ff99e7aa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:00.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.796+0000 7ff99e7aa700 1 Processor -- start 2023-07-02T09:03:00.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.796+0000 7ff99e7aa700 1 -- start start 2023-07-02T09:03:00.797 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff99e7aa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff99809a220 0x7ff9981348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:00.797 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff99e7aa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99809cea0 con 0x7ff99806d6e0 2023-07-02T09:03:00.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff99e7aa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99806d490 con 0x7ff99809a220 2023-07-02T09:03:00.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff9977fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff99809a220 0x7ff9981348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:00.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff997fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff99806d6e0 0x7ff998099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42632/0 (socket says 172.21.15.103:42632) 2023-07-02T09:03:00.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff997fff700 1 -- 172.21.15.103:0/2480330412 learned_addr learned my addr 172.21.15.103:0/2480330412 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:00.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff9977fe700 1 -- 172.21.15.103:0/2480330412 >> v1:172.21.15.103:6789/0 conn(0x7ff99806d6e0 legacy=0x7ff998099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:00.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff9977fe700 1 -- 172.21.15.103:0/2480330412 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99806dc80 con 0x7ff99809a220 2023-07-02T09:03:00.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff996ffd700 1 -- 172.21.15.103:0/2480330412 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000562414 0 0) 0x7ff99809cea0 con 0x7ff99806d6e0 2023-07-02T09:03:00.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff9977fe700 1 --2- 172.21.15.103:0/2480330412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99809a220 0x7ff9981348e0 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7ff0b7d52c7b52 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:00.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff996ffd700 1 -- 172.21.15.103:0/2480330412 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff98c010440 con 0x7ff99809a220 2023-07-02T09:03:00.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff996ffd700 1 -- 172.21.15.103:0/2480330412 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff98c0105a0 con 0x7ff99809a220 2023-07-02T09:03:00.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.797+0000 7ff996ffd700 1 -- 172.21.15.103:0/2480330412 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff98c010840 con 0x7ff99809a220 2023-07-02T09:03:00.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.798+0000 7ff99e7aa700 1 -- 172.21.15.103:0/2480330412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99809a220 msgr2=0x7ff9981348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:00.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.798+0000 7ff99e7aa700 1 --2- 172.21.15.103:0/2480330412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99809a220 0x7ff9981348e0 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:00.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.798+0000 7ff99e7aa700 1 -- 172.21.15.103:0/2480330412 shutdown_connections 2023-07-02T09:03:00.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.798+0000 7ff99e7aa700 1 --2- 172.21.15.103:0/2480330412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99809a220 0x7ff9981348e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:00.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.798+0000 7ff99e7aa700 1 -- 172.21.15.103:0/2480330412 >> 172.21.15.103:0/2480330412 conn(0x7ff99809d810 msgr2=0x7ff99809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:00.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.798+0000 7ff99e7aa700 1 -- 172.21.15.103:0/2480330412 shutdown_connections 2023-07-02T09:03:00.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.798+0000 7ff99e7aa700 1 -- 172.21.15.103:0/2480330412 wait complete. 2023-07-02T09:03:00.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.798+0000 7ff99e7aa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:00.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.799+0000 7ff99e7aa700 1 Processor -- start 2023-07-02T09:03:00.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.799+0000 7ff99e7aa700 1 -- start start 2023-07-02T09:03:00.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.800+0000 7ff99e7aa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99806d6e0 0x7ff99811d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:00.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.800+0000 7ff99e7aa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff998120a00 con 0x7ff99806d6e0 2023-07-02T09:03:00.803 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.800+0000 7ff997fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99806d6e0 0x7ff99811d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:00.803 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.800+0000 7ff997fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99806d6e0 0x7ff99811d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56288/0 (socket says 172.21.15.103:56288) 2023-07-02T09:03:00.803 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.800+0000 7ff997fff700 1 -- 172.21.15.103:0/3812469988 learned_addr learned my addr 172.21.15.103:0/3812469988 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:00.803 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.800+0000 7ff997fff700 1 -- 172.21.15.103:0/3812469988 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99811c790 con 0x7ff99806d6e0 2023-07-02T09:03:00.803 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.801+0000 7ff997fff700 1 --2- 172.21.15.103:0/3812469988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99806d6e0 0x7ff99811d090 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:00.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.801+0000 7ff994ff9700 1 -- 172.21.15.103:0/3812469988 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff988004b80 con 0x7ff99806d6e0 2023-07-02T09:03:00.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.801+0000 7ff994ff9700 1 -- 172.21.15.103:0/3812469988 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff988004ce0 con 0x7ff99806d6e0 2023-07-02T09:03:00.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.801+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99811cd00 con 0x7ff99806d6e0 2023-07-02T09:03:00.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.801+0000 7ff994ff9700 1 -- 172.21.15.103:0/3812469988 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff98800c550 con 0x7ff99806d6e0 2023-07-02T09:03:00.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.801+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99811e3f0 con 0x7ff99806d6e0 2023-07-02T09:03:00.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.802+0000 7ff994ff9700 1 -- 172.21.15.103:0/3812469988 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ff98800c6b0 con 0x7ff99806d6e0 2023-07-02T09:03:00.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.802+0000 7ff994ff9700 1 --2- 172.21.15.103:0/3812469988 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff9840377c0 0x7ff984039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:00.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.802+0000 7ff994ff9700 1 -- 172.21.15.103:0/3812469988 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ff9880469e0 con 0x7ff99806d6e0 2023-07-02T09:03:00.805 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.802+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff998061ea0 con 0x7ff99806d6e0 2023-07-02T09:03:00.805 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.802+0000 7ff9977fe700 1 --2- 172.21.15.103:0/3812469988 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff9840377c0 0x7ff984039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:00.805 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.803+0000 7ff9977fe700 1 --2- 172.21.15.103:0/3812469988 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff9840377c0 0x7ff984039cd0 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:00.807 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.807+0000 7ff994ff9700 1 -- 172.21.15.103:0/3812469988 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ff988009c80 con 0x7ff99806d6e0 2023-07-02T09:03:00.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.995+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff998073ca0 con 0x7ff99806d6e0 2023-07-02T09:03:00.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.995+0000 7ff994ff9700 1 -- 172.21.15.103:0/3812469988 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ff9880098a0 con 0x7ff99806d6e0 2023-07-02T09:03:00.995 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:00.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff9840377c0 msgr2=0x7ff984039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:00.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 --2- 172.21.15.103:0/3812469988 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff9840377c0 0x7ff984039cd0 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:00.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99806d6e0 msgr2=0x7ff99811d090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:00.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 --2- 172.21.15.103:0/3812469988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99806d6e0 0x7ff99811d090 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:00.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 shutdown_connections 2023-07-02T09:03:00.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 --2- 172.21.15.103:0/3812469988 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff9840377c0 0x7ff984039cd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:00.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 --2- 172.21.15.103:0/3812469988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff99806d6e0 0x7ff99811d090 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:00.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 >> 172.21.15.103:0/3812469988 conn(0x7ff99809d810 msgr2=0x7ff998070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:00.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 shutdown_connections 2023-07-02T09:03:00.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:00.997+0000 7ff99e7aa700 1 -- 172.21.15.103:0/3812469988 wait complete. 2023-07-02T09:03:01.010 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:02.011 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:02.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.150+0000 7f4a4ecc0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:02.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.155+0000 7f4a4ecc0700 1 Processor -- start 2023-07-02T09:03:02.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.155+0000 7f4a4ecc0700 1 -- start start 2023-07-02T09:03:02.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a4ecc0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4a4806d6e0 0x7f4a48099b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:02.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a4ecc0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4809cea0 con 0x7f4a4809a050 2023-07-02T09:03:02.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a4ecc0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4806d490 con 0x7f4a4806d6e0 2023-07-02T09:03:02.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a4ca5c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4a4806d6e0 0x7f4a48099b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:02.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a47fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4a4809a050 0x7f4a4812f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:42642/0 (socket says 172.21.15.103:42642) 2023-07-02T09:03:02.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a47fff700 1 -- 172.21.15.103:0/3404270169 learned_addr learned my addr 172.21.15.103:0/3404270169 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:02.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a4ca5c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4a4806d6e0 0x7f4a48099b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56294/0 (socket says 172.21.15.103:56294) 2023-07-02T09:03:02.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a4ca5c700 1 -- 172.21.15.103:0/3404270169 >> v1:172.21.15.103:6789/0 conn(0x7f4a4809a050 legacy=0x7f4a4812f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:02.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a4ca5c700 1 -- 172.21.15.103:0/3404270169 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4806dc80 con 0x7f4a4806d6e0 2023-07-02T09:03:02.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a477fe700 1 -- 172.21.15.103:0/3404270169 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874281089 0 0) 0x7f4a4809cea0 con 0x7f4a4809a050 2023-07-02T09:03:02.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a4ca5c700 1 --2- 172.21.15.103:0/3404270169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a48099b10 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae9ba28e1d42eb4 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:02.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a477fe700 1 -- 172.21.15.103:0/3404270169 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4a3c010440 con 0x7f4a4806d6e0 2023-07-02T09:03:02.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a477fe700 1 -- 172.21.15.103:0/3404270169 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4a3c0105a0 con 0x7f4a4806d6e0 2023-07-02T09:03:02.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.156+0000 7f4a477fe700 1 -- 172.21.15.103:0/3404270169 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4a3c010840 con 0x7f4a4806d6e0 2023-07-02T09:03:02.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.157+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/3404270169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 msgr2=0x7f4a48099b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:02.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.157+0000 7f4a4ecc0700 1 --2- 172.21.15.103:0/3404270169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a48099b10 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:02.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.157+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/3404270169 shutdown_connections 2023-07-02T09:03:02.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.157+0000 7f4a4ecc0700 1 --2- 172.21.15.103:0/3404270169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a48099b10 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:02.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.157+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/3404270169 >> 172.21.15.103:0/3404270169 conn(0x7f4a4809d810 msgr2=0x7f4a4809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:02.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.157+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/3404270169 shutdown_connections 2023-07-02T09:03:02.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.157+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/3404270169 wait complete. 2023-07-02T09:03:02.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.157+0000 7f4a4ecc0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:02.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.159+0000 7f4a4ecc0700 1 Processor -- start 2023-07-02T09:03:02.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.159+0000 7f4a4ecc0700 1 -- start start 2023-07-02T09:03:02.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ecc0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a481c2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:02.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ecc0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4812efc0 con 0x7f4a4806d6e0 2023-07-02T09:03:02.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ca5c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a481c2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:02.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ca5c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a481c2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56296/0 (socket says 172.21.15.103:56296) 2023-07-02T09:03:02.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ca5c700 1 -- 172.21.15.103:0/4002402561 learned_addr learned my addr 172.21.15.103:0/4002402561 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:02.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ca5c700 1 -- 172.21.15.103:0/4002402561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4811fc40 con 0x7f4a4806d6e0 2023-07-02T09:03:02.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ca5c700 1 --2- 172.21.15.103:0/4002402561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a481c2c50 crc :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:02.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a457fa700 1 -- 172.21.15.103:0/4002402561 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4a3c00fa50 con 0x7f4a4806d6e0 2023-07-02T09:03:02.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a457fa700 1 -- 172.21.15.103:0/4002402561 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4a3c016040 con 0x7f4a4806d6e0 2023-07-02T09:03:02.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a457fa700 1 -- 172.21.15.103:0/4002402561 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4a3c00fce0 con 0x7f4a4806d6e0 2023-07-02T09:03:02.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4811ec70 con 0x7f4a4806d6e0 2023-07-02T09:03:02.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.160+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a481b5750 con 0x7f4a4806d6e0 2023-07-02T09:03:02.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.161+0000 7f4a457fa700 1 -- 172.21.15.103:0/4002402561 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4a3c00e360 con 0x7f4a4806d6e0 2023-07-02T09:03:02.164 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.161+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a48061ea0 con 0x7f4a4806d6e0 2023-07-02T09:03:02.164 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.162+0000 7f4a457fa700 1 --2- 172.21.15.103:0/4002402561 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a2c0377c0 0x7f4a2c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:02.164 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.162+0000 7f4a457fa700 1 -- 172.21.15.103:0/4002402561 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4a3c04a8d0 con 0x7f4a4806d6e0 2023-07-02T09:03:02.164 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.162+0000 7f4a47fff700 1 --2- 172.21.15.103:0/4002402561 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a2c0377c0 0x7f4a2c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:02.165 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.163+0000 7f4a47fff700 1 --2- 172.21.15.103:0/4002402561 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a2c0377c0 0x7f4a2c039cd0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:02.166 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.166+0000 7f4a457fa700 1 -- 172.21.15.103:0/4002402561 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f4a3c017220 con 0x7f4a4806d6e0 2023-07-02T09:03:02.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.352+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4a4811fc40 con 0x7f4a4806d6e0 2023-07-02T09:03:02.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.353+0000 7f4a457fa700 1 -- 172.21.15.103:0/4002402561 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4a4811fc40 con 0x7f4a4806d6e0 2023-07-02T09:03:02.353 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:02.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.354+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a2c0377c0 msgr2=0x7f4a2c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:02.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.354+0000 7f4a4ecc0700 1 --2- 172.21.15.103:0/4002402561 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a2c0377c0 0x7f4a2c039cd0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:02.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.355+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 msgr2=0x7f4a481c2c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:02.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.355+0000 7f4a4ecc0700 1 --2- 172.21.15.103:0/4002402561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a481c2c50 crc :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:02.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.355+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 shutdown_connections 2023-07-02T09:03:02.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.355+0000 7f4a4ecc0700 1 --2- 172.21.15.103:0/4002402561 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a2c0377c0 0x7f4a2c039cd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:02.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.355+0000 7f4a4ecc0700 1 --2- 172.21.15.103:0/4002402561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a4806d6e0 0x7f4a481c2c50 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:02.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.355+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 >> 172.21.15.103:0/4002402561 conn(0x7f4a4809d810 msgr2=0x7f4a48070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:02.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.355+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 shutdown_connections 2023-07-02T09:03:02.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:02.355+0000 7f4a4ecc0700 1 -- 172.21.15.103:0/4002402561 wait complete. 2023-07-02T09:03:02.368 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:03.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:03.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.509+0000 7f4aaabc6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:03.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.514+0000 7f4aaabc6700 1 Processor -- start 2023-07-02T09:03:03.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.514+0000 7f4aaabc6700 1 -- start start 2023-07-02T09:03:03.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aaabc6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4aa4132f50 0x7f4aa4133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:03.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aaabc6700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa409e260 con 0x7f4aa409c580 2023-07-02T09:03:03.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aaabc6700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa40a1640 con 0x7f4aa4132f50 2023-07-02T09:03:03.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aa8962700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4aa4132f50 0x7f4aa4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:03.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aa8962700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4aa4132f50 0x7f4aa4133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44258/0 (socket says 172.21.15.103:44258) 2023-07-02T09:03:03.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aa3fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4aa409c580 0x7f4aa409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46786/0 (socket says 172.21.15.103:46786) 2023-07-02T09:03:03.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aa8962700 1 -- 172.21.15.103:0/2757610116 learned_addr learned my addr 172.21.15.103:0/2757610116 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:03.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aa8962700 1 -- 172.21.15.103:0/2757610116 >> v1:172.21.15.103:6789/0 conn(0x7f4aa409c580 legacy=0x7f4aa409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:03.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aa8962700 1 -- 172.21.15.103:0/2757610116 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa4132a20 con 0x7f4aa4132f50 2023-07-02T09:03:03.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aa8962700 1 --2- 172.21.15.103:0/2757610116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa4132f50 0x7f4aa4133360 crc :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ff658e94cd9fc9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:03.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.515+0000 7f4aa37fe700 1 -- 172.21.15.103:0/2757610116 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4a98009d90 con 0x7f4aa4132f50 2023-07-02T09:03:03.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aa37fe700 1 -- 172.21.15.103:0/2757610116 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4a9800e230 con 0x7f4aa4132f50 2023-07-02T09:03:03.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aa37fe700 1 -- 172.21.15.103:0/2757610116 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4a9800e4d0 con 0x7f4aa4132f50 2023-07-02T09:03:03.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aaabc6700 1 -- 172.21.15.103:0/2757610116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa4132f50 msgr2=0x7f4aa4133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:03.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aaabc6700 1 --2- 172.21.15.103:0/2757610116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa4132f50 0x7f4aa4133360 crc :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:03.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aaabc6700 1 -- 172.21.15.103:0/2757610116 shutdown_connections 2023-07-02T09:03:03.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aaabc6700 1 --2- 172.21.15.103:0/2757610116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa4132f50 0x7f4aa4133360 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:03.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aaabc6700 1 -- 172.21.15.103:0/2757610116 >> 172.21.15.103:0/2757610116 conn(0x7f4aa40a2080 msgr2=0x7f4aa40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:03.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aaabc6700 1 -- 172.21.15.103:0/2757610116 shutdown_connections 2023-07-02T09:03:03.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.516+0000 7f4aaabc6700 1 -- 172.21.15.103:0/2757610116 wait complete. 2023-07-02T09:03:03.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.517+0000 7f4aaabc6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:03.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.517+0000 7f4aaabc6700 1 Processor -- start 2023-07-02T09:03:03.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.518+0000 7f4aaabc6700 1 -- start start 2023-07-02T09:03:03.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.518+0000 7f4aaabc6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa409c580 0x7f4aa412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:03.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.518+0000 7f4aaabc6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa406f9d0 con 0x7f4aa409c580 2023-07-02T09:03:03.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.518+0000 7f4aa8962700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa409c580 0x7f4aa412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:03.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.518+0000 7f4aa8962700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa409c580 0x7f4aa412b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44266/0 (socket says 172.21.15.103:44266) 2023-07-02T09:03:03.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.518+0000 7f4aa8962700 1 -- 172.21.15.103:0/3982257432 learned_addr learned my addr 172.21.15.103:0/3982257432 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:03.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.519+0000 7f4aa8962700 1 -- 172.21.15.103:0/3982257432 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa4133a00 con 0x7f4aa409c580 2023-07-02T09:03:03.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.519+0000 7f4aa8962700 1 --2- 172.21.15.103:0/3982257432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa409c580 0x7f4aa412b7b0 crc :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:03.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.519+0000 7f4aa17fa700 1 -- 172.21.15.103:0/3982257432 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4a98014540 con 0x7f4aa409c580 2023-07-02T09:03:03.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.519+0000 7f4aa17fa700 1 -- 172.21.15.103:0/3982257432 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4a980146a0 con 0x7f4aa409c580 2023-07-02T09:03:03.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.519+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa406d160 con 0x7f4aa409c580 2023-07-02T09:03:03.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.519+0000 7f4aa17fa700 1 -- 172.21.15.103:0/3982257432 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4a98014940 con 0x7f4aa409c580 2023-07-02T09:03:03.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.519+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa406da70 con 0x7f4aa409c580 2023-07-02T09:03:03.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.520+0000 7f4aa17fa700 1 -- 172.21.15.103:0/3982257432 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4a98014af0 con 0x7f4aa409c580 2023-07-02T09:03:03.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.520+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4061ea0 con 0x7f4aa409c580 2023-07-02T09:03:03.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.520+0000 7f4aa17fa700 1 --2- 172.21.15.103:0/3982257432 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a880377b0 0x7f4a88039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:03.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.520+0000 7f4aa17fa700 1 -- 172.21.15.103:0/3982257432 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4a980482d0 con 0x7f4aa409c580 2023-07-02T09:03:03.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.520+0000 7f4aa3fff700 1 --2- 172.21.15.103:0/3982257432 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a880377b0 0x7f4a88039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:03.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.521+0000 7f4aa3fff700 1 --2- 172.21.15.103:0/3982257432 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a880377b0 0x7f4a88039cc0 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:03.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.525+0000 7f4aa17fa700 1 -- 172.21.15.103:0/3982257432 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f4a9800d1c0 con 0x7f4aa409c580 2023-07-02T09:03:03.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.714+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4aa40725b0 con 0x7f4aa409c580 2023-07-02T09:03:03.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.714+0000 7f4aa17fa700 1 -- 172.21.15.103:0/3982257432 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4a98011320 con 0x7f4aa409c580 2023-07-02T09:03:03.715 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:03.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a880377b0 msgr2=0x7f4a88039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:03.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 --2- 172.21.15.103:0/3982257432 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a880377b0 0x7f4a88039cc0 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:03.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa409c580 msgr2=0x7f4aa412b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:03.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 --2- 172.21.15.103:0/3982257432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa409c580 0x7f4aa412b7b0 crc :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:03.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 shutdown_connections 2023-07-02T09:03:03.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 --2- 172.21.15.103:0/3982257432 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4a880377b0 0x7f4a88039cc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:03.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 --2- 172.21.15.103:0/3982257432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4aa409c580 0x7f4aa412b7b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:03.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 >> 172.21.15.103:0/3982257432 conn(0x7f4aa40a2080 msgr2=0x7f4aa41babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:03.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 shutdown_connections 2023-07-02T09:03:03.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:03.716+0000 7f4aaabc6700 1 -- 172.21.15.103:0/3982257432 wait complete. 2023-07-02T09:03:03.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:04.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~198s 2023-07-02T09:03:04.730 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:04.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.869+0000 7f307392e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:04.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.873+0000 7f307392e700 1 Processor -- start 2023-07-02T09:03:04.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.874+0000 7f307392e700 1 -- start start 2023-07-02T09:03:04.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.874+0000 7f307392e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f306c06d6e0 0x7f306c099b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:04.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.874+0000 7f307392e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306c09cea0 con 0x7f306c09a050 2023-07-02T09:03:04.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.874+0000 7f307392e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306c06d490 con 0x7f306c06d6e0 2023-07-02T09:03:04.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f30716ca700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f306c06d6e0 0x7f306c099b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:04.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f3070ec9700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f306c09a050 0x7f306c12f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46800/0 (socket says 172.21.15.103:46800) 2023-07-02T09:03:04.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f30716ca700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f306c06d6e0 0x7f306c099b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44272/0 (socket says 172.21.15.103:44272) 2023-07-02T09:03:04.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f3070ec9700 1 -- 172.21.15.103:0/568514092 learned_addr learned my addr 172.21.15.103:0/568514092 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:04.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f30716ca700 1 -- 172.21.15.103:0/568514092 >> v1:172.21.15.103:6789/0 conn(0x7f306c09a050 legacy=0x7f306c12f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:04.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f30716ca700 1 -- 172.21.15.103:0/568514092 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c06dc80 con 0x7f306c06d6e0 2023-07-02T09:03:04.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f305ffff700 1 -- 172.21.15.103:0/568514092 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002776996 0 0) 0x7f306c09cea0 con 0x7f306c09a050 2023-07-02T09:03:04.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f30716ca700 1 --2- 172.21.15.103:0/568514092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c099b10 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a9e53f496cce3d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:04.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f305ffff700 1 -- 172.21.15.103:0/568514092 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3068009cd0 con 0x7f306c06d6e0 2023-07-02T09:03:04.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f305ffff700 1 -- 172.21.15.103:0/568514092 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3068009e30 con 0x7f306c06d6e0 2023-07-02T09:03:04.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.875+0000 7f305ffff700 1 -- 172.21.15.103:0/568514092 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f306800e360 con 0x7f306c06d6e0 2023-07-02T09:03:04.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.876+0000 7f307392e700 1 -- 172.21.15.103:0/568514092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 msgr2=0x7f306c099b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:04.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.876+0000 7f307392e700 1 --2- 172.21.15.103:0/568514092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c099b10 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:04.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.876+0000 7f307392e700 1 -- 172.21.15.103:0/568514092 shutdown_connections 2023-07-02T09:03:04.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.876+0000 7f307392e700 1 --2- 172.21.15.103:0/568514092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c099b10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:04.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.876+0000 7f307392e700 1 -- 172.21.15.103:0/568514092 >> 172.21.15.103:0/568514092 conn(0x7f306c09d810 msgr2=0x7f306c09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:04.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.876+0000 7f307392e700 1 -- 172.21.15.103:0/568514092 shutdown_connections 2023-07-02T09:03:04.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.876+0000 7f307392e700 1 -- 172.21.15.103:0/568514092 wait complete. 2023-07-02T09:03:04.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.876+0000 7f307392e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:04.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.877+0000 7f307392e700 1 Processor -- start 2023-07-02T09:03:04.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.877+0000 7f307392e700 1 -- start start 2023-07-02T09:03:04.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f307392e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c06f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:04.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f307392e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c06f0d0 con 0x7f306c06d6e0 2023-07-02T09:03:04.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f30716ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c06f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:04.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f30716ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c06f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44286/0 (socket says 172.21.15.103:44286) 2023-07-02T09:03:04.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f30716ca700 1 -- 172.21.15.103:0/1793594616 learned_addr learned my addr 172.21.15.103:0/1793594616 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:04.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f30716ca700 1 -- 172.21.15.103:0/1793594616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c06ef10 con 0x7f306c06d6e0 2023-07-02T09:03:04.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f30716ca700 1 --2- 172.21.15.103:0/1793594616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c06f490 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:04.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f305dffb700 1 -- 172.21.15.103:0/1793594616 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3068013530 con 0x7f306c06d6e0 2023-07-02T09:03:04.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f305dffb700 1 -- 172.21.15.103:0/1793594616 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3068013690 con 0x7f306c06d6e0 2023-07-02T09:03:04.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f305dffb700 1 -- 172.21.15.103:0/1793594616 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3068013930 con 0x7f306c06d6e0 2023-07-02T09:03:04.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.878+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306c123ff0 con 0x7f306c06d6e0 2023-07-02T09:03:04.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.879+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306c124dd0 con 0x7f306c06d6e0 2023-07-02T09:03:04.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.879+0000 7f305dffb700 1 -- 172.21.15.103:0/1793594616 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3068013ae0 con 0x7f306c06d6e0 2023-07-02T09:03:04.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.879+0000 7f305dffb700 1 --2- 172.21.15.103:0/1793594616 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30500377b0 0x7f3050039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:04.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.879+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306c070dd0 con 0x7f306c06d6e0 2023-07-02T09:03:04.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.880+0000 7f305dffb700 1 -- 172.21.15.103:0/1793594616 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3068018040 con 0x7f306c06d6e0 2023-07-02T09:03:04.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.880+0000 7f3070ec9700 1 --2- 172.21.15.103:0/1793594616 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30500377b0 0x7f3050039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:04.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.880+0000 7f3070ec9700 1 --2- 172.21.15.103:0/1793594616 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30500377b0 0x7f3050039cc0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:04.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:04.884+0000 7f305dffb700 1 -- 172.21.15.103:0/1793594616 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3068018a80 con 0x7f306c06d6e0 2023-07-02T09:03:05.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.071+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f306c06ef10 con 0x7f306c06d6e0 2023-07-02T09:03:05.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.072+0000 7f305dffb700 1 -- 172.21.15.103:0/1793594616 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3068018670 con 0x7f306c06d6e0 2023-07-02T09:03:05.072 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:05.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30500377b0 msgr2=0x7f3050039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:05.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 --2- 172.21.15.103:0/1793594616 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30500377b0 0x7f3050039cc0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:05.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 msgr2=0x7f306c06f490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:05.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 --2- 172.21.15.103:0/1793594616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c06f490 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:05.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 shutdown_connections 2023-07-02T09:03:05.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 --2- 172.21.15.103:0/1793594616 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f30500377b0 0x7f3050039cc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:05.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 --2- 172.21.15.103:0/1793594616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f306c06d6e0 0x7f306c06f490 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:05.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 >> 172.21.15.103:0/1793594616 conn(0x7f306c09d810 msgr2=0x7f306c09f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:05.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 shutdown_connections 2023-07-02T09:03:05.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:05.074+0000 7f307392e700 1 -- 172.21.15.103:0/1793594616 wait complete. 2023-07-02T09:03:05.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:06.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:06.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.229+0000 7faf70a3a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:06.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.234+0000 7faf70a3a700 1 Processor -- start 2023-07-02T09:03:06.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.234+0000 7faf70a3a700 1 -- start start 2023-07-02T09:03:06.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf70a3a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf6c133360 0x7faf6c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:06.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf70a3a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c09e260 con 0x7faf6c132f50 2023-07-02T09:03:06.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf70a3a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6c0a1640 con 0x7faf6c133360 2023-07-02T09:03:06.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf69d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf6c133360 0x7faf6c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:06.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf6a59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7faf6c132f50 0x7faf6c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46802/0 (socket says 172.21.15.103:46802) 2023-07-02T09:03:06.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf6a59c700 1 -- 172.21.15.103:0/2812956691 learned_addr learned my addr 172.21.15.103:0/2812956691 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:06.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf69d9b700 1 --2- 172.21.15.103:0/2812956691 >> v2:172.21.15.103:3300/0 conn(0x7faf6c133360 0x7faf6c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44298/0 (socket says 172.21.15.103:44298) 2023-07-02T09:03:06.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf69d9b700 1 -- 172.21.15.103:0/2812956691 >> v1:172.21.15.103:6789/0 conn(0x7faf6c132f50 legacy=0x7faf6c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:06.236 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf69d9b700 1 -- 172.21.15.103:0/2812956691 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c132a20 con 0x7faf6c133360 2023-07-02T09:03:06.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf6959a700 1 -- 172.21.15.103:0/2812956691 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835190674 0 0) 0x7faf6c09e260 con 0x7faf6c132f50 2023-07-02T09:03:06.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf69d9b700 1 --2- 172.21.15.103:0/2812956691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c133360 0x7faf6c09cc90 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ab5708c9121df9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:06.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf6959a700 1 -- 172.21.15.103:0/2812956691 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf5c009cd0 con 0x7faf6c133360 2023-07-02T09:03:06.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf6959a700 1 -- 172.21.15.103:0/2812956691 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7faf5c009e30 con 0x7faf6c133360 2023-07-02T09:03:06.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.235+0000 7faf6959a700 1 -- 172.21.15.103:0/2812956691 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf5c00e360 con 0x7faf6c133360 2023-07-02T09:03:06.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.236+0000 7faf70a3a700 1 -- 172.21.15.103:0/2812956691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c133360 msgr2=0x7faf6c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:06.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.236+0000 7faf70a3a700 1 --2- 172.21.15.103:0/2812956691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c133360 0x7faf6c09cc90 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:06.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.236+0000 7faf70a3a700 1 -- 172.21.15.103:0/2812956691 shutdown_connections 2023-07-02T09:03:06.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.236+0000 7faf70a3a700 1 --2- 172.21.15.103:0/2812956691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c133360 0x7faf6c09cc90 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:06.239 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.236+0000 7faf70a3a700 1 -- 172.21.15.103:0/2812956691 >> 172.21.15.103:0/2812956691 conn(0x7faf6c0a2080 msgr2=0x7faf6c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:06.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.236+0000 7faf70a3a700 1 -- 172.21.15.103:0/2812956691 shutdown_connections 2023-07-02T09:03:06.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.236+0000 7faf70a3a700 1 -- 172.21.15.103:0/2812956691 wait complete. 2023-07-02T09:03:06.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.236+0000 7faf70a3a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:06.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.237+0000 7faf70a3a700 1 Processor -- start 2023-07-02T09:03:06.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.237+0000 7faf70a3a700 1 -- start start 2023-07-02T09:03:06.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.238+0000 7faf70a3a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c132f50 0x7faf6c1ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:06.240 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.238+0000 7faf70a3a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c06d370 con 0x7faf6c132f50 2023-07-02T09:03:06.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.238+0000 7faf6a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c132f50 0x7faf6c1ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:06.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.238+0000 7faf6a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c132f50 0x7faf6c1ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44314/0 (socket says 172.21.15.103:44314) 2023-07-02T09:03:06.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.238+0000 7faf6a59c700 1 -- 172.21.15.103:0/1643696045 learned_addr learned my addr 172.21.15.103:0/1643696045 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:06.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.238+0000 7faf6a59c700 1 -- 172.21.15.103:0/1643696045 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c06dde0 con 0x7faf6c132f50 2023-07-02T09:03:06.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.239+0000 7faf6a59c700 1 --2- 172.21.15.103:0/1643696045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c132f50 0x7faf6c1ba9b0 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:06.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.239+0000 7faf5affd700 1 -- 172.21.15.103:0/1643696045 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf60004b80 con 0x7faf6c132f50 2023-07-02T09:03:06.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.239+0000 7faf5affd700 1 -- 172.21.15.103:0/1643696045 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7faf60004ce0 con 0x7faf6c132f50 2023-07-02T09:03:06.241 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.239+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c1ba4f0 con 0x7faf6c132f50 2023-07-02T09:03:06.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.239+0000 7faf5affd700 1 -- 172.21.15.103:0/1643696045 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7faf6000c4f0 con 0x7faf6c132f50 2023-07-02T09:03:06.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.239+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c071e40 con 0x7faf6c132f50 2023-07-02T09:03:06.242 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.240+0000 7faf5affd700 1 -- 172.21.15.103:0/1643696045 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7faf6000c650 con 0x7faf6c132f50 2023-07-02T09:03:06.243 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.240+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c061ea0 con 0x7faf6c132f50 2023-07-02T09:03:06.243 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.240+0000 7faf5affd700 1 --2- 172.21.15.103:0/1643696045 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf4c0377c0 0x7faf4c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:06.243 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.240+0000 7faf5affd700 1 -- 172.21.15.103:0/1643696045 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7faf60046970 con 0x7faf6c132f50 2023-07-02T09:03:06.243 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.240+0000 7faf69d9b700 1 --2- 172.21.15.103:0/1643696045 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf4c0377c0 0x7faf4c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:06.243 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.241+0000 7faf69d9b700 1 --2- 172.21.15.103:0/1643696045 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf4c0377c0 0x7faf4c039cd0 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.245+0000 7faf5affd700 1 -- 172.21.15.103:0/1643696045 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7faf60009c80 con 0x7faf6c132f50 2023-07-02T09:03:06.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.432+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7faf6c0725b0 con 0x7faf6c132f50 2023-07-02T09:03:06.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.433+0000 7faf5affd700 1 -- 172.21.15.103:0/1643696045 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7faf600098a0 con 0x7faf6c132f50 2023-07-02T09:03:06.433 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:06.435 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf4c0377c0 msgr2=0x7faf4c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:06.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 --2- 172.21.15.103:0/1643696045 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf4c0377c0 0x7faf4c039cd0 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:06.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c132f50 msgr2=0x7faf6c1ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:06.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 --2- 172.21.15.103:0/1643696045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c132f50 0x7faf6c1ba9b0 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:06.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 shutdown_connections 2023-07-02T09:03:06.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 --2- 172.21.15.103:0/1643696045 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7faf4c0377c0 0x7faf4c039cd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:06.436 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 --2- 172.21.15.103:0/1643696045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf6c132f50 0x7faf6c1ba9b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:06.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 >> 172.21.15.103:0/1643696045 conn(0x7faf6c0a2080 msgr2=0x7faf6c0a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:06.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 shutdown_connections 2023-07-02T09:03:06.437 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:06.435+0000 7faf70a3a700 1 -- 172.21.15.103:0/1643696045 wait complete. 2023-07-02T09:03:06.448 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:07.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:07.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.582+0000 7f40268e7700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:07.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f40268e7700 1 Processor -- start 2023-07-02T09:03:07.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f40268e7700 1 -- start start 2023-07-02T09:03:07.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f40268e7700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4020132f50 0x7f4020133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:07.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f40268e7700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402009e260 con 0x7f402009c580 2023-07-02T09:03:07.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f40268e7700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40200a1640 con 0x7f4020132f50 2023-07-02T09:03:07.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f401ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4020132f50 0x7f4020133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:07.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f401ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4020132f50 0x7f4020133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44330/0 (socket says 172.21.15.103:44330) 2023-07-02T09:03:07.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f401f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f402009c580 0x7f402009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46808/0 (socket says 172.21.15.103:46808) 2023-07-02T09:03:07.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.588+0000 7f401ffff700 1 -- 172.21.15.103:0/1214520602 learned_addr learned my addr 172.21.15.103:0/1214520602 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:07.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.589+0000 7f401ffff700 1 -- 172.21.15.103:0/1214520602 >> v1:172.21.15.103:6789/0 conn(0x7f402009c580 legacy=0x7f402009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:07.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.589+0000 7f401ffff700 1 -- 172.21.15.103:0/1214520602 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020132a20 con 0x7f4020132f50 2023-07-02T09:03:07.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.589+0000 7f401effd700 1 -- 172.21.15.103:0/1214520602 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473606632 0 0) 0x7f402009e260 con 0x7f402009c580 2023-07-02T09:03:07.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.589+0000 7f401ffff700 1 --2- 172.21.15.103:0/1214520602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4020132f50 0x7f4020133360 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b1fcdddbd9bb89 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:07.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.589+0000 7f401effd700 1 -- 172.21.15.103:0/1214520602 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4014009d90 con 0x7f4020132f50 2023-07-02T09:03:07.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.589+0000 7f401effd700 1 -- 172.21.15.103:0/1214520602 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f401400e230 con 0x7f4020132f50 2023-07-02T09:03:07.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.589+0000 7f401effd700 1 -- 172.21.15.103:0/1214520602 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f401400e4d0 con 0x7f4020132f50 2023-07-02T09:03:07.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.590+0000 7f40268e7700 1 -- 172.21.15.103:0/1214520602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4020132f50 msgr2=0x7f4020133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:07.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.590+0000 7f40268e7700 1 --2- 172.21.15.103:0/1214520602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4020132f50 0x7f4020133360 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:07.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.590+0000 7f40268e7700 1 -- 172.21.15.103:0/1214520602 shutdown_connections 2023-07-02T09:03:07.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.590+0000 7f40268e7700 1 --2- 172.21.15.103:0/1214520602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4020132f50 0x7f4020133360 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:07.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.590+0000 7f40268e7700 1 -- 172.21.15.103:0/1214520602 >> 172.21.15.103:0/1214520602 conn(0x7f40200a2080 msgr2=0x7f40200a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:07.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.590+0000 7f40268e7700 1 -- 172.21.15.103:0/1214520602 shutdown_connections 2023-07-02T09:03:07.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.590+0000 7f40268e7700 1 -- 172.21.15.103:0/1214520602 wait complete. 2023-07-02T09:03:07.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.590+0000 7f40268e7700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:07.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.591+0000 7f40268e7700 1 Processor -- start 2023-07-02T09:03:07.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.591+0000 7f40268e7700 1 -- start start 2023-07-02T09:03:07.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.592+0000 7f40268e7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402009c580 0x7f40201baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:07.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.592+0000 7f40268e7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40201ba610 con 0x7f402009c580 2023-07-02T09:03:07.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.592+0000 7f401ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402009c580 0x7f40201baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:07.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.592+0000 7f401ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402009c580 0x7f40201baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44336/0 (socket says 172.21.15.103:44336) 2023-07-02T09:03:07.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.592+0000 7f401ffff700 1 -- 172.21.15.103:0/2811921476 learned_addr learned my addr 172.21.15.103:0/2811921476 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:07.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.593+0000 7f401ffff700 1 -- 172.21.15.103:0/2811921476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402006dfc0 con 0x7f402009c580 2023-07-02T09:03:07.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.593+0000 7f401ffff700 1 --2- 172.21.15.103:0/2811921476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402009c580 0x7f40201baa60 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:07.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.593+0000 7f401cff9700 1 -- 172.21.15.103:0/2811921476 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4014014410 con 0x7f402009c580 2023-07-02T09:03:07.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.593+0000 7f401cff9700 1 -- 172.21.15.103:0/2811921476 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4014014570 con 0x7f402009c580 2023-07-02T09:03:07.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.593+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40201ba270 con 0x7f402009c580 2023-07-02T09:03:07.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.593+0000 7f401cff9700 1 -- 172.21.15.103:0/2811921476 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4014014810 con 0x7f402009c580 2023-07-02T09:03:07.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.593+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f402006dba0 con 0x7f402009c580 2023-07-02T09:03:07.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.594+0000 7f401cff9700 1 -- 172.21.15.103:0/2811921476 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f40140149c0 con 0x7f402009c580 2023-07-02T09:03:07.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.594+0000 7f401cff9700 1 --2- 172.21.15.103:0/2811921476 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f400c0377c0 0x7f400c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:07.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.594+0000 7f401cff9700 1 -- 172.21.15.103:0/2811921476 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f40140482f0 con 0x7f402009c580 2023-07-02T09:03:07.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.594+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40201ba460 con 0x7f402009c580 2023-07-02T09:03:07.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.594+0000 7f401f7fe700 1 --2- 172.21.15.103:0/2811921476 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f400c0377c0 0x7f400c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:07.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.595+0000 7f401f7fe700 1 --2- 172.21.15.103:0/2811921476 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f400c0377c0 0x7f400c039cd0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:07.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.599+0000 7f401cff9700 1 -- 172.21.15.103:0/2811921476 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f401400fb20 con 0x7f402009c580 2023-07-02T09:03:07.792 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.792+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4020061ea0 con 0x7f402009c580 2023-07-02T09:03:07.793 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.792+0000 7f401cff9700 1 -- 172.21.15.103:0/2811921476 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f401400f710 con 0x7f402009c580 2023-07-02T09:03:07.793 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:07.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f400c0377c0 msgr2=0x7f400c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:07.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 --2- 172.21.15.103:0/2811921476 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f400c0377c0 0x7f400c039cd0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:07.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402009c580 msgr2=0x7f40201baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:07.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 --2- 172.21.15.103:0/2811921476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402009c580 0x7f40201baa60 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:07.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 shutdown_connections 2023-07-02T09:03:07.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 --2- 172.21.15.103:0/2811921476 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f400c0377c0 0x7f400c039cd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:07.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 --2- 172.21.15.103:0/2811921476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402009c580 0x7f40201baa60 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:07.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 >> 172.21.15.103:0/2811921476 conn(0x7f40200a2080 msgr2=0x7f4020130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:07.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.794+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 shutdown_connections 2023-07-02T09:03:07.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:07.795+0000 7f40268e7700 1 -- 172.21.15.103:0/2811921476 wait complete. 2023-07-02T09:03:07.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:08.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:08.952 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.948+0000 7f7bd95bc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:08.953 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.953+0000 7f7bd95bc700 1 Processor -- start 2023-07-02T09:03:08.953 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.953+0000 7f7bd95bc700 1 -- start start 2023-07-02T09:03:08.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd95bc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7bd4132030 0x7f7bd4134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:08.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd95bc700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd406df60 con 0x7f7bd412f250 2023-07-02T09:03:08.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd95bc700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd409a710 con 0x7f7bd4132030 2023-07-02T09:03:08.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7bd4132030 0x7f7bd4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:08.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd2ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7bd412f250 0x7f7bd412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46820/0 (socket says 172.21.15.103:46820) 2023-07-02T09:03:08.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd2ffd700 1 -- 172.21.15.103:0/1710599194 learned_addr learned my addr 172.21.15.103:0/1710599194 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:08.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd27fc700 1 -- 172.21.15.103:0/1710599194 >> v1:172.21.15.103:6789/0 conn(0x7f7bd412f250 legacy=0x7f7bd412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:08.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd27fc700 1 -- 172.21.15.103:0/1710599194 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd412ed20 con 0x7f7bd4132030 2023-07-02T09:03:08.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd1ffb700 1 -- 172.21.15.103:0/1710599194 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814578859 0 0) 0x7f7bd406df60 con 0x7f7bd412f250 2023-07-02T09:03:08.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd27fc700 1 --2- 172.21.15.103:0/1710599194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd4132030 0x7f7bd4134450 crc :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559fa99273ee5a46 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:08.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd1ffb700 1 -- 172.21.15.103:0/1710599194 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7bc400d450 con 0x7f7bd4132030 2023-07-02T09:03:08.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd1ffb700 1 -- 172.21.15.103:0/1710599194 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7bc400d5b0 con 0x7f7bd4132030 2023-07-02T09:03:08.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.954+0000 7f7bd1ffb700 1 -- 172.21.15.103:0/1710599194 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7bc400d850 con 0x7f7bd4132030 2023-07-02T09:03:08.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.955+0000 7f7bd95bc700 1 -- 172.21.15.103:0/1710599194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd4132030 msgr2=0x7f7bd4134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:08.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.955+0000 7f7bd95bc700 1 --2- 172.21.15.103:0/1710599194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd4132030 0x7f7bd4134450 crc :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:08.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.955+0000 7f7bd95bc700 1 -- 172.21.15.103:0/1710599194 shutdown_connections 2023-07-02T09:03:08.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.955+0000 7f7bd95bc700 1 --2- 172.21.15.103:0/1710599194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd4132030 0x7f7bd4134450 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:08.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.955+0000 7f7bd95bc700 1 -- 172.21.15.103:0/1710599194 >> 172.21.15.103:0/1710599194 conn(0x7f7bd409d810 msgr2=0x7f7bd409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:08.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.955+0000 7f7bd95bc700 1 -- 172.21.15.103:0/1710599194 shutdown_connections 2023-07-02T09:03:08.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.955+0000 7f7bd95bc700 1 -- 172.21.15.103:0/1710599194 wait complete. 2023-07-02T09:03:08.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.955+0000 7f7bd95bc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:08.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.957+0000 7f7bd95bc700 1 Processor -- start 2023-07-02T09:03:08.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.957+0000 7f7bd95bc700 1 -- start start 2023-07-02T09:03:08.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd95bc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd412f250 0x7f7bd406ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:08.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd95bc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd41175d0 con 0x7f7bd412f250 2023-07-02T09:03:08.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd412f250 0x7f7bd406ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:08.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd412f250 0x7f7bd406ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44346/0 (socket says 172.21.15.103:44346) 2023-07-02T09:03:08.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd2ffd700 1 -- 172.21.15.103:0/3854420309 learned_addr learned my addr 172.21.15.103:0/3854420309 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:08.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd2ffd700 1 -- 172.21.15.103:0/3854420309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd4119080 con 0x7f7bd412f250 2023-07-02T09:03:08.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd2ffd700 1 --2- 172.21.15.103:0/3854420309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd412f250 0x7f7bd406ff70 crc :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:08.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bbb7fe700 1 -- 172.21.15.103:0/3854420309 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7bc8004b80 con 0x7f7bd412f250 2023-07-02T09:03:08.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bbb7fe700 1 -- 172.21.15.103:0/3854420309 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7bc8004ce0 con 0x7f7bd412f250 2023-07-02T09:03:08.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bbb7fe700 1 -- 172.21.15.103:0/3854420309 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7bc800c4f0 con 0x7f7bd412f250 2023-07-02T09:03:08.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd41292f0 con 0x7f7bd412f250 2023-07-02T09:03:08.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.958+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd4117af0 con 0x7f7bd412f250 2023-07-02T09:03:08.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.959+0000 7f7bbb7fe700 1 -- 172.21.15.103:0/3854420309 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7bc800c650 con 0x7f7bd412f250 2023-07-02T09:03:08.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.959+0000 7f7bbb7fe700 1 --2- 172.21.15.103:0/3854420309 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7bc00377c0 0x7f7bc0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:08.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.959+0000 7f7bbb7fe700 1 -- 172.21.15.103:0/3854420309 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7bc8046970 con 0x7f7bd412f250 2023-07-02T09:03:08.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.960+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd4061ea0 con 0x7f7bd412f250 2023-07-02T09:03:08.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.960+0000 7f7bd27fc700 1 --2- 172.21.15.103:0/3854420309 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7bc00377c0 0x7f7bc0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:08.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.960+0000 7f7bd27fc700 1 --2- 172.21.15.103:0/3854420309 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7bc00377c0 0x7f7bc0039cd0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:08.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:08.965+0000 7f7bbb7fe700 1 -- 172.21.15.103:0/3854420309 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7bc8009c80 con 0x7f7bd412f250 2023-07-02T09:03:09.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.154+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7bd409cce0 con 0x7f7bd412f250 2023-07-02T09:03:09.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.154+0000 7f7bbb7fe700 1 -- 172.21.15.103:0/3854420309 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7bc802c020 con 0x7f7bd412f250 2023-07-02T09:03:09.155 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:09.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7bc00377c0 msgr2=0x7f7bc0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:09.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 --2- 172.21.15.103:0/3854420309 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7bc00377c0 0x7f7bc0039cd0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:09.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd412f250 msgr2=0x7f7bd406ff70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:09.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 --2- 172.21.15.103:0/3854420309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd412f250 0x7f7bd406ff70 crc :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:09.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 shutdown_connections 2023-07-02T09:03:09.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 --2- 172.21.15.103:0/3854420309 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7bc00377c0 0x7f7bc0039cd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:09.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 --2- 172.21.15.103:0/3854420309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd412f250 0x7f7bd406ff70 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:09.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 >> 172.21.15.103:0/3854420309 conn(0x7f7bd409d810 msgr2=0x7f7bd412e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:09.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 shutdown_connections 2023-07-02T09:03:09.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:09.156+0000 7f7bd95bc700 1 -- 172.21.15.103:0/3854420309 wait complete. 2023-07-02T09:03:09.169 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:09.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~204s 2023-07-02T09:03:10.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:10.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.310+0000 7f3e41049700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:10.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.315+0000 7f3e41049700 1 Processor -- start 2023-07-02T09:03:10.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.315+0000 7f3e41049700 1 -- start start 2023-07-02T09:03:10.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e41049700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3e3c133360 0x7f3e3c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:10.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e41049700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3c09e260 con 0x7f3e3c132f50 2023-07-02T09:03:10.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e41049700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c0a1640 con 0x7f3e3c133360 2023-07-02T09:03:10.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e3a59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3e3c133360 0x7f3e3c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:10.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e3ad9d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3e3c132f50 0x7f3e3c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46832/0 (socket says 172.21.15.103:46832) 2023-07-02T09:03:10.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e3a59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3e3c133360 0x7f3e3c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44350/0 (socket says 172.21.15.103:44350) 2023-07-02T09:03:10.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e3ad9d700 1 -- 172.21.15.103:0/101277508 learned_addr learned my addr 172.21.15.103:0/101277508 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:10.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e3a59c700 1 -- 172.21.15.103:0/101277508 >> v1:172.21.15.103:6789/0 conn(0x7f3e3c132f50 legacy=0x7f3e3c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:10.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e3a59c700 1 -- 172.21.15.103:0/101277508 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c132a20 con 0x7f3e3c133360 2023-07-02T09:03:10.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e3a59c700 1 --2- 172.21.15.103:0/101277508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c133360 0x7f3e3c09cc90 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f061030d209e5176 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:10.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e39d9b700 1 -- 172.21.15.103:0/101277508 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3e30009cd0 con 0x7f3e3c133360 2023-07-02T09:03:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e39d9b700 1 -- 172.21.15.103:0/101277508 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3e30009e30 con 0x7f3e3c133360 2023-07-02T09:03:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.316+0000 7f3e39d9b700 1 -- 172.21.15.103:0/101277508 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3e3000e360 con 0x7f3e3c133360 2023-07-02T09:03:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.317+0000 7f3e41049700 1 -- 172.21.15.103:0/101277508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c133360 msgr2=0x7f3e3c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.317+0000 7f3e41049700 1 --2- 172.21.15.103:0/101277508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c133360 0x7f3e3c09cc90 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.317+0000 7f3e41049700 1 -- 172.21.15.103:0/101277508 shutdown_connections 2023-07-02T09:03:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.317+0000 7f3e41049700 1 --2- 172.21.15.103:0/101277508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c133360 0x7f3e3c09cc90 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.317+0000 7f3e41049700 1 -- 172.21.15.103:0/101277508 >> 172.21.15.103:0/101277508 conn(0x7f3e3c0a2080 msgr2=0x7f3e3c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.317+0000 7f3e41049700 1 -- 172.21.15.103:0/101277508 shutdown_connections 2023-07-02T09:03:10.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.317+0000 7f3e41049700 1 -- 172.21.15.103:0/101277508 wait complete. 2023-07-02T09:03:10.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.317+0000 7f3e41049700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:10.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.319+0000 7f3e41049700 1 Processor -- start 2023-07-02T09:03:10.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.319+0000 7f3e41049700 1 -- start start 2023-07-02T09:03:10.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.319+0000 7f3e41049700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c132f50 0x7f3e3c12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:10.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.319+0000 7f3e41049700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c06d260 con 0x7f3e3c132f50 2023-07-02T09:03:10.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.319+0000 7f3e3ad9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c132f50 0x7f3e3c12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:10.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.319+0000 7f3e3ad9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c132f50 0x7f3e3c12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44364/0 (socket says 172.21.15.103:44364) 2023-07-02T09:03:10.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.319+0000 7f3e3ad9d700 1 -- 172.21.15.103:0/3285259414 learned_addr learned my addr 172.21.15.103:0/3285259414 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:10.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.320+0000 7f3e3ad9d700 1 -- 172.21.15.103:0/3285259414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c12bb90 con 0x7f3e3c132f50 2023-07-02T09:03:10.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.320+0000 7f3e3ad9d700 1 --2- 172.21.15.103:0/3285259414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c132f50 0x7f3e3c12bea0 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:10.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.320+0000 7f3e237fe700 1 -- 172.21.15.103:0/3285259414 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3e2c005a60 con 0x7f3e3c132f50 2023-07-02T09:03:10.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.320+0000 7f3e237fe700 1 -- 172.21.15.103:0/3285259414 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3e2c005bc0 con 0x7f3e3c132f50 2023-07-02T09:03:10.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.320+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c06f730 con 0x7f3e3c132f50 2023-07-02T09:03:10.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.320+0000 7f3e237fe700 1 -- 172.21.15.103:0/3285259414 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3e2c005e60 con 0x7f3e3c132f50 2023-07-02T09:03:10.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.320+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c06f490 con 0x7f3e3c132f50 2023-07-02T09:03:10.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.321+0000 7f3e237fe700 1 -- 172.21.15.103:0/3285259414 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f3e2c0070b0 con 0x7f3e3c132f50 2023-07-02T09:03:10.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.321+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c061ea0 con 0x7f3e3c132f50 2023-07-02T09:03:10.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.321+0000 7f3e237fe700 1 --2- 172.21.15.103:0/3285259414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3e280377c0 0x7f3e28039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:10.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.321+0000 7f3e237fe700 1 -- 172.21.15.103:0/3285259414 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3e2c0469f0 con 0x7f3e3c132f50 2023-07-02T09:03:10.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.321+0000 7f3e3a59c700 1 --2- 172.21.15.103:0/3285259414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3e280377c0 0x7f3e28039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:10.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.322+0000 7f3e3a59c700 1 --2- 172.21.15.103:0/3285259414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3e280377c0 0x7f3e28039cd0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:10.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.326+0000 7f3e237fe700 1 -- 172.21.15.103:0/3285259414 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f3e2c00a4a0 con 0x7f3e3c132f50 2023-07-02T09:03:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.511+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3e3c06f490 con 0x7f3e3c132f50 2023-07-02T09:03:10.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.511+0000 7f3e237fe700 1 -- 172.21.15.103:0/3285259414 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f3e3c06f490 con 0x7f3e3c132f50 2023-07-02T09:03:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3e280377c0 msgr2=0x7f3e28039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 --2- 172.21.15.103:0/3285259414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3e280377c0 0x7f3e28039cd0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c132f50 msgr2=0x7f3e3c12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 --2- 172.21.15.103:0/3285259414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c132f50 0x7f3e3c12bea0 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 shutdown_connections 2023-07-02T09:03:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 --2- 172.21.15.103:0/3285259414 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f3e280377c0 0x7f3e28039cd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 --2- 172.21.15.103:0/3285259414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e3c132f50 0x7f3e3c12bea0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:10.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 >> 172.21.15.103:0/3285259414 conn(0x7f3e3c0a2080 msgr2=0x7f3e3c1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:10.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 shutdown_connections 2023-07-02T09:03:10.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:10.513+0000 7f3e41049700 1 -- 172.21.15.103:0/3285259414 wait complete. 2023-07-02T09:03:10.526 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:11.527 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:11.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.666+0000 7f7a6642b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:11.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.671+0000 7f7a6642b700 1 Processor -- start 2023-07-02T09:03:11.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.671+0000 7f7a6642b700 1 -- start start 2023-07-02T09:03:11.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a6642b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7a60132f50 0x7f7a60133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:11.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a6642b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6009e260 con 0x7f7a6009c580 2023-07-02T09:03:11.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a6642b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a600a1640 con 0x7f7a60132f50 2023-07-02T09:03:11.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a5ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7a60132f50 0x7f7a60133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:11.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a5f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7a6009c580 0x7f7a6009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46836/0 (socket says 172.21.15.103:46836) 2023-07-02T09:03:11.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a5ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7a60132f50 0x7f7a60133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44374/0 (socket says 172.21.15.103:44374) 2023-07-02T09:03:11.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a5f7fe700 1 -- 172.21.15.103:0/3585784446 learned_addr learned my addr 172.21.15.103:0/3585784446 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:11.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a5ffff700 1 -- 172.21.15.103:0/3585784446 >> v1:172.21.15.103:6789/0 conn(0x7f7a6009c580 legacy=0x7f7a6009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:11.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a5ffff700 1 -- 172.21.15.103:0/3585784446 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a60132a20 con 0x7f7a60132f50 2023-07-02T09:03:11.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.672+0000 7f7a5ffff700 1 --2- 172.21.15.103:0/3585784446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a60132f50 0x7f7a60133360 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e28337749f774d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:11.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a5effd700 1 -- 172.21.15.103:0/3585784446 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a54009cd0 con 0x7f7a60132f50 2023-07-02T09:03:11.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a5effd700 1 -- 172.21.15.103:0/3585784446 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7a54009e30 con 0x7f7a60132f50 2023-07-02T09:03:11.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a5effd700 1 -- 172.21.15.103:0/3585784446 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a5400e360 con 0x7f7a60132f50 2023-07-02T09:03:11.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a6642b700 1 -- 172.21.15.103:0/3585784446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a60132f50 msgr2=0x7f7a60133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:11.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a6642b700 1 --2- 172.21.15.103:0/3585784446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a60132f50 0x7f7a60133360 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:11.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a6642b700 1 -- 172.21.15.103:0/3585784446 shutdown_connections 2023-07-02T09:03:11.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a6642b700 1 --2- 172.21.15.103:0/3585784446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a60132f50 0x7f7a60133360 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:11.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a6642b700 1 -- 172.21.15.103:0/3585784446 >> 172.21.15.103:0/3585784446 conn(0x7f7a600a2080 msgr2=0x7f7a600a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:11.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a6642b700 1 -- 172.21.15.103:0/3585784446 shutdown_connections 2023-07-02T09:03:11.676 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.673+0000 7f7a6642b700 1 -- 172.21.15.103:0/3585784446 wait complete. 2023-07-02T09:03:11.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.674+0000 7f7a6642b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:11.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.674+0000 7f7a6642b700 1 Processor -- start 2023-07-02T09:03:11.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.675+0000 7f7a6642b700 1 -- start start 2023-07-02T09:03:11.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.675+0000 7f7a6642b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a6009c580 0x7f7a6012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:11.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.675+0000 7f7a6642b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6006f9d0 con 0x7f7a6009c580 2023-07-02T09:03:11.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.675+0000 7f7a5ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a6009c580 0x7f7a6012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:11.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a5ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a6009c580 0x7f7a6012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44378/0 (socket says 172.21.15.103:44378) 2023-07-02T09:03:11.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a5ffff700 1 -- 172.21.15.103:0/708906216 learned_addr learned my addr 172.21.15.103:0/708906216 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:11.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a5ffff700 1 -- 172.21.15.103:0/708906216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a60133a00 con 0x7f7a6009c580 2023-07-02T09:03:11.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a5ffff700 1 --2- 172.21.15.103:0/708906216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a6009c580 0x7f7a6012b7b0 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:11.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a5cff9700 1 -- 172.21.15.103:0/708906216 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a54013530 con 0x7f7a6009c580 2023-07-02T09:03:11.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a5cff9700 1 -- 172.21.15.103:0/708906216 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7a54013690 con 0x7f7a6009c580 2023-07-02T09:03:11.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a5cff9700 1 -- 172.21.15.103:0/708906216 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7a54013930 con 0x7f7a6009c580 2023-07-02T09:03:11.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6006d160 con 0x7f7a6009c580 2023-07-02T09:03:11.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.676+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6006da70 con 0x7f7a6009c580 2023-07-02T09:03:11.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.677+0000 7f7a5cff9700 1 -- 172.21.15.103:0/708906216 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7a54013ae0 con 0x7f7a6009c580 2023-07-02T09:03:11.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.677+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a60061ea0 con 0x7f7a6009c580 2023-07-02T09:03:11.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.677+0000 7f7a5cff9700 1 --2- 172.21.15.103:0/708906216 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a4c0377c0 0x7f7a4c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:11.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.677+0000 7f7a5cff9700 1 -- 172.21.15.103:0/708906216 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7a54018040 con 0x7f7a6009c580 2023-07-02T09:03:11.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.677+0000 7f7a5f7fe700 1 --2- 172.21.15.103:0/708906216 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a4c0377c0 0x7f7a4c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:11.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.678+0000 7f7a5f7fe700 1 --2- 172.21.15.103:0/708906216 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a4c0377c0 0x7f7a4c039cd0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:11.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.682+0000 7f7a5cff9700 1 -- 172.21.15.103:0/708906216 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7a54018dd0 con 0x7f7a6009c580 2023-07-02T09:03:11.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.869+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7a600725b0 con 0x7f7a6009c580 2023-07-02T09:03:11.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.870+0000 7f7a5cff9700 1 -- 172.21.15.103:0/708906216 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7a540189f0 con 0x7f7a6009c580 2023-07-02T09:03:11.870 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:11.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a4c0377c0 msgr2=0x7f7a4c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:11.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 --2- 172.21.15.103:0/708906216 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a4c0377c0 0x7f7a4c039cd0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:11.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a6009c580 msgr2=0x7f7a6012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:11.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 --2- 172.21.15.103:0/708906216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a6009c580 0x7f7a6012b7b0 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:11.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 shutdown_connections 2023-07-02T09:03:11.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 --2- 172.21.15.103:0/708906216 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7a4c0377c0 0x7f7a4c039cd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:11.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 --2- 172.21.15.103:0/708906216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a6009c580 0x7f7a6012b7b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:11.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 >> 172.21.15.103:0/708906216 conn(0x7f7a600a2080 msgr2=0x7f7a601babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:11.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 shutdown_connections 2023-07-02T09:03:11.874 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:11.872+0000 7f7a6642b700 1 -- 172.21.15.103:0/708906216 wait complete. 2023-07-02T09:03:11.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:12.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:13.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.023+0000 7fb764d16700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:13.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.028+0000 7fb764d16700 1 Processor -- start 2023-07-02T09:03:13.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.028+0000 7fb764d16700 1 -- start start 2023-07-02T09:03:13.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb764d16700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb76012fb10 0x7fb76012ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb764d16700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76012f750 con 0x7fb76006bf60 2023-07-02T09:03:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb764d16700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76009abe0 con 0x7fb76012fb10 2023-07-02T09:03:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75e59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb76012fb10 0x7fb76012ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75e59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb76012fb10 0x7fb76012ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44388/0 (socket says 172.21.15.103:44388) 2023-07-02T09:03:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75dd9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb76006bf60 0x7fb76006c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46850/0 (socket says 172.21.15.103:46850) 2023-07-02T09:03:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75e59c700 1 -- 172.21.15.103:0/2982261866 learned_addr learned my addr 172.21.15.103:0/2982261866 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75e59c700 1 -- 172.21.15.103:0/2982261866 >> v1:172.21.15.103:6789/0 conn(0x7fb76006bf60 legacy=0x7fb76006c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75e59c700 1 -- 172.21.15.103:0/2982261866 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb760066840 con 0x7fb76012fb10 2023-07-02T09:03:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75d59a700 1 -- 172.21.15.103:0/2982261866 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698017147 0 0) 0x7fb76012f750 con 0x7fb76006bf60 2023-07-02T09:03:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75e59c700 1 --2- 172.21.15.103:0/2982261866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76012fb10 0x7fb76012ff20 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306743b2557e9e0c server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75d59a700 1 -- 172.21.15.103:0/2982261866 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb750009d90 con 0x7fb76012fb10 2023-07-02T09:03:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75d59a700 1 -- 172.21.15.103:0/2982261866 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb7500059f0 con 0x7fb76012fb10 2023-07-02T09:03:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.029+0000 7fb75d59a700 1 -- 172.21.15.103:0/2982261866 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb750005c90 con 0x7fb76012fb10 2023-07-02T09:03:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.030+0000 7fb764d16700 1 -- 172.21.15.103:0/2982261866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76012fb10 msgr2=0x7fb76012ff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.030+0000 7fb764d16700 1 --2- 172.21.15.103:0/2982261866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76012fb10 0x7fb76012ff20 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.030+0000 7fb764d16700 1 -- 172.21.15.103:0/2982261866 shutdown_connections 2023-07-02T09:03:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.030+0000 7fb764d16700 1 --2- 172.21.15.103:0/2982261866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76012fb10 0x7fb76012ff20 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.030+0000 7fb764d16700 1 -- 172.21.15.103:0/2982261866 >> 172.21.15.103:0/2982261866 conn(0x7fb7600a0100 msgr2=0x7fb7600a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.030+0000 7fb764d16700 1 -- 172.21.15.103:0/2982261866 shutdown_connections 2023-07-02T09:03:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.030+0000 7fb764d16700 1 -- 172.21.15.103:0/2982261866 wait complete. 2023-07-02T09:03:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.030+0000 7fb764d16700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.032+0000 7fb764d16700 1 Processor -- start 2023-07-02T09:03:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.032+0000 7fb764d16700 1 -- start start 2023-07-02T09:03:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb764d16700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76006bf60 0x7fb7601c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb764d16700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb760121b90 con 0x7fb76006bf60 2023-07-02T09:03:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb75e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76006bf60 0x7fb7601c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb75e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76006bf60 0x7fb7601c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44402/0 (socket says 172.21.15.103:44402) 2023-07-02T09:03:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb75e59c700 1 -- 172.21.15.103:0/3429471751 learned_addr learned my addr 172.21.15.103:0/3429471751 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb75e59c700 1 -- 172.21.15.103:0/3429471751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76011fce0 con 0x7fb76006bf60 2023-07-02T09:03:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb75e59c700 1 --2- 172.21.15.103:0/3429471751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76006bf60 0x7fb7601c6de0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:13.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb74effd700 1 -- 172.21.15.103:0/3429471751 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb750006900 con 0x7fb76006bf60 2023-07-02T09:03:13.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb74effd700 1 -- 172.21.15.103:0/3429471751 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb750013340 con 0x7fb76006bf60 2023-07-02T09:03:13.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76012f350 con 0x7fb76006bf60 2023-07-02T09:03:13.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.033+0000 7fb74effd700 1 -- 172.21.15.103:0/3429471751 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb7500135e0 con 0x7fb76006bf60 2023-07-02T09:03:13.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.034+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76011f3b0 con 0x7fb76006bf60 2023-07-02T09:03:13.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.034+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76004e800 con 0x7fb76006bf60 2023-07-02T09:03:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.035+0000 7fb74effd700 1 -- 172.21.15.103:0/3429471751 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb750013790 con 0x7fb76006bf60 2023-07-02T09:03:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.035+0000 7fb74effd700 1 --2- 172.21.15.103:0/3429471751 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7400377c0 0x7fb740039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.035+0000 7fb74effd700 1 -- 172.21.15.103:0/3429471751 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb750048340 con 0x7fb76006bf60 2023-07-02T09:03:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.036+0000 7fb75dd9b700 1 --2- 172.21.15.103:0/3429471751 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7400377c0 0x7fb740039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:13.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.036+0000 7fb75dd9b700 1 --2- 172.21.15.103:0/3429471751 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7400377c0 0x7fb740039cd0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:13.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.040+0000 7fb74effd700 1 -- 172.21.15.103:0/3429471751 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb75000f8f0 con 0x7fb76006bf60 2023-07-02T09:03:13.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.229+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb76012f1c0 con 0x7fb76006bf60 2023-07-02T09:03:13.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.230+0000 7fb74effd700 1 -- 172.21.15.103:0/3429471751 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb75000f710 con 0x7fb76006bf60 2023-07-02T09:03:13.230 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:13.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7400377c0 msgr2=0x7fb740039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:13.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 --2- 172.21.15.103:0/3429471751 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7400377c0 0x7fb740039cd0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:13.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76006bf60 msgr2=0x7fb7601c6de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:13.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 --2- 172.21.15.103:0/3429471751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76006bf60 0x7fb7601c6de0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:13.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 shutdown_connections 2023-07-02T09:03:13.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 --2- 172.21.15.103:0/3429471751 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7400377c0 0x7fb740039cd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:13.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 --2- 172.21.15.103:0/3429471751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb76006bf60 0x7fb7601c6de0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:13.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 >> 172.21.15.103:0/3429471751 conn(0x7fb7600a0100 msgr2=0x7fb760132aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:13.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 shutdown_connections 2023-07-02T09:03:13.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:13.232+0000 7fb764d16700 1 -- 172.21.15.103:0/3429471751 wait complete. 2023-07-02T09:03:13.245 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:14.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:14.389 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.384+0000 7fa2f99aa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:14.390 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.389+0000 7fa2f99aa700 1 Processor -- start 2023-07-02T09:03:14.390 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.389+0000 7fa2f99aa700 1 -- start start 2023-07-02T09:03:14.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f99aa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa2f409a220 0x7fa2f41348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:14.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f99aa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f409cea0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f99aa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f406d490 con 0x7fa2f409a220 2023-07-02T09:03:14.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa2f409a220 0x7fa2f41348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:14.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f2ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa2f406d6e0 0x7fa2f4099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55794/0 (socket says 172.21.15.103:55794) 2023-07-02T09:03:14.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f2ffd700 1 -- 172.21.15.103:0/2194073200 learned_addr learned my addr 172.21.15.103:0/2194073200 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:14.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f27fc700 1 -- 172.21.15.103:0/2194073200 >> v1:172.21.15.103:6789/0 conn(0x7fa2f406d6e0 legacy=0x7fa2f4099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:14.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f27fc700 1 -- 172.21.15.103:0/2194073200 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f406dc80 con 0x7fa2f409a220 2023-07-02T09:03:14.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f1ffb700 1 -- 172.21.15.103:0/2194073200 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595195965 0 0) 0x7fa2f409cea0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f27fc700 1 --2- 172.21.15.103:0/2194073200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f409a220 0x7fa2f41348e0 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b865fdc0fe57e720 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:14.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f1ffb700 1 -- 172.21.15.103:0/2194073200 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa2e4009cd0 con 0x7fa2f409a220 2023-07-02T09:03:14.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f1ffb700 1 -- 172.21.15.103:0/2194073200 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa2e4009e30 con 0x7fa2f409a220 2023-07-02T09:03:14.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.390+0000 7fa2f1ffb700 1 -- 172.21.15.103:0/2194073200 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa2e400e360 con 0x7fa2f409a220 2023-07-02T09:03:14.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.391+0000 7fa2f99aa700 1 -- 172.21.15.103:0/2194073200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f409a220 msgr2=0x7fa2f41348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:14.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.391+0000 7fa2f99aa700 1 --2- 172.21.15.103:0/2194073200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f409a220 0x7fa2f41348e0 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:14.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.391+0000 7fa2f99aa700 1 -- 172.21.15.103:0/2194073200 shutdown_connections 2023-07-02T09:03:14.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.391+0000 7fa2f99aa700 1 --2- 172.21.15.103:0/2194073200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f409a220 0x7fa2f41348e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:14.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.391+0000 7fa2f99aa700 1 -- 172.21.15.103:0/2194073200 >> 172.21.15.103:0/2194073200 conn(0x7fa2f409d810 msgr2=0x7fa2f409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:14.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.391+0000 7fa2f99aa700 1 -- 172.21.15.103:0/2194073200 shutdown_connections 2023-07-02T09:03:14.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.391+0000 7fa2f99aa700 1 -- 172.21.15.103:0/2194073200 wait complete. 2023-07-02T09:03:14.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.391+0000 7fa2f99aa700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:14.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.392+0000 7fa2f99aa700 1 Processor -- start 2023-07-02T09:03:14.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.393+0000 7fa2f99aa700 1 -- start start 2023-07-02T09:03:14.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.393+0000 7fa2f99aa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f406d6e0 0x7fa2f411dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:14.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.393+0000 7fa2f99aa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4121a60 con 0x7fa2f406d6e0 2023-07-02T09:03:14.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.393+0000 7fa2f2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f406d6e0 0x7fa2f411dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:14.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.393+0000 7fa2f2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f406d6e0 0x7fa2f411dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48460/0 (socket says 172.21.15.103:48460) 2023-07-02T09:03:14.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.393+0000 7fa2f2ffd700 1 -- 172.21.15.103:0/1394649659 learned_addr learned my addr 172.21.15.103:0/1394649659 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:14.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.394+0000 7fa2f2ffd700 1 -- 172.21.15.103:0/1394649659 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f411d7f0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.394+0000 7fa2f2ffd700 1 --2- 172.21.15.103:0/1394649659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f406d6e0 0x7fa2f411dea0 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.394+0000 7fa2dbfff700 1 -- 172.21.15.103:0/1394649659 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa2e8004b80 con 0x7fa2f406d6e0 2023-07-02T09:03:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.394+0000 7fa2dbfff700 1 -- 172.21.15.103:0/1394649659 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa2e8004d50 con 0x7fa2f406d6e0 2023-07-02T09:03:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.394+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f411db00 con 0x7fa2f406d6e0 2023-07-02T09:03:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.394+0000 7fa2dbfff700 1 -- 172.21.15.103:0/1394649659 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa2e80070c0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.394+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f412f280 con 0x7fa2f406d6e0 2023-07-02T09:03:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.395+0000 7fa2dbfff700 1 -- 172.21.15.103:0/1394649659 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa2e8007270 con 0x7fa2f406d6e0 2023-07-02T09:03:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.395+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f411dcf0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.395+0000 7fa2dbfff700 1 --2- 172.21.15.103:0/1394649659 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2e0037770 0x7fa2e0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:14.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.395+0000 7fa2dbfff700 1 -- 172.21.15.103:0/1394649659 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa2e80497a0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.395+0000 7fa2f27fc700 1 --2- 172.21.15.103:0/1394649659 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2e0037770 0x7fa2e0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:14.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.396+0000 7fa2f27fc700 1 --2- 172.21.15.103:0/1394649659 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2e0037770 0x7fa2e0039c80 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:14.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.400+0000 7fa2dbfff700 1 -- 172.21.15.103:0/1394649659 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa2e80014d0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.586+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa2f4061ea0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.586+0000 7fa2dbfff700 1 -- 172.21.15.103:0/1394649659 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa2e80010e0 con 0x7fa2f406d6e0 2023-07-02T09:03:14.588 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:14.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2e0037770 msgr2=0x7fa2e0039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 --2- 172.21.15.103:0/1394649659 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2e0037770 0x7fa2e0039c80 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f406d6e0 msgr2=0x7fa2f411dea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 --2- 172.21.15.103:0/1394649659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f406d6e0 0x7fa2f411dea0 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 shutdown_connections 2023-07-02T09:03:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 --2- 172.21.15.103:0/1394649659 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2e0037770 0x7fa2e0039c80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 --2- 172.21.15.103:0/1394649659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f406d6e0 0x7fa2f411dea0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:14.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 >> 172.21.15.103:0/1394649659 conn(0x7fa2f409d810 msgr2=0x7fa2f4117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:14.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 shutdown_connections 2023-07-02T09:03:14.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:14.588+0000 7fa2f99aa700 1 -- 172.21.15.103:0/1394649659 wait complete. 2023-07-02T09:03:14.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:15.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~209s 2023-07-02T09:03:15.603 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:15.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.743+0000 7f0ea2700700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:15.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.748+0000 7f0ea2700700 1 Processor -- start 2023-07-02T09:03:15.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.748+0000 7f0ea2700700 1 -- start start 2023-07-02T09:03:15.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.749+0000 7f0ea2700700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0e9c133880 0x7f0e9c135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:15.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.749+0000 7f0ea2700700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c071cc0 con 0x7f0e9c097520 2023-07-02T09:03:15.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.749+0000 7f0ea2700700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9c071270 con 0x7f0e9c133880 2023-07-02T09:03:15.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.749+0000 7f0e9b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0e9c133880 0x7f0e9c135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:15.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.749+0000 7f0e9bfff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f0e9c097520 0x7f0e9c097930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55798/0 (socket says 172.21.15.103:55798) 2023-07-02T09:03:15.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.749+0000 7f0e9b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0e9c133880 0x7f0e9c135ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48474/0 (socket says 172.21.15.103:48474) 2023-07-02T09:03:15.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.749+0000 7f0e9bfff700 1 -- 172.21.15.103:0/3270978654 learned_addr learned my addr 172.21.15.103:0/3270978654 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:15.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.749+0000 7f0e9b7fe700 1 -- 172.21.15.103:0/3270978654 >> v1:172.21.15.103:6789/0 conn(0x7f0e9c097520 legacy=0x7f0e9c097930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0e9b7fe700 1 -- 172.21.15.103:0/3270978654 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c09d1c0 con 0x7f0e9c133880 2023-07-02T09:03:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0e9affd700 1 -- 172.21.15.103:0/3270978654 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703202545 0 0) 0x7f0e9c071cc0 con 0x7f0e9c097520 2023-07-02T09:03:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0e9b7fe700 1 --2- 172.21.15.103:0/3270978654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c133880 0x7f0e9c135ca0 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed116ff873824dd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0e9affd700 1 -- 172.21.15.103:0/3270978654 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0e8c009cd0 con 0x7f0e9c133880 2023-07-02T09:03:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0e9affd700 1 -- 172.21.15.103:0/3270978654 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0e8c009e30 con 0x7f0e9c133880 2023-07-02T09:03:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0e9affd700 1 -- 172.21.15.103:0/3270978654 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0e8c00e360 con 0x7f0e9c133880 2023-07-02T09:03:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0ea2700700 1 -- 172.21.15.103:0/3270978654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c133880 msgr2=0x7f0e9c135ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0ea2700700 1 --2- 172.21.15.103:0/3270978654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c133880 0x7f0e9c135ca0 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0ea2700700 1 -- 172.21.15.103:0/3270978654 shutdown_connections 2023-07-02T09:03:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0ea2700700 1 --2- 172.21.15.103:0/3270978654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c133880 0x7f0e9c135ca0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.750+0000 7f0ea2700700 1 -- 172.21.15.103:0/3270978654 >> 172.21.15.103:0/3270978654 conn(0x7f0e9c0a1030 msgr2=0x7f0e9c0a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.751+0000 7f0ea2700700 1 -- 172.21.15.103:0/3270978654 shutdown_connections 2023-07-02T09:03:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.751+0000 7f0ea2700700 1 -- 172.21.15.103:0/3270978654 wait complete. 2023-07-02T09:03:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.751+0000 7f0ea2700700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:15.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.752+0000 7f0ea2700700 1 Processor -- start 2023-07-02T09:03:15.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.752+0000 7f0ea2700700 1 -- start start 2023-07-02T09:03:15.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.752+0000 7f0ea2700700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c097520 0x7f0e9c1c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:15.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.752+0000 7f0ea2700700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c1c4c40 con 0x7f0e9c097520 2023-07-02T09:03:15.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0e9bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c097520 0x7f0e9c1c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:15.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0e9bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c097520 0x7f0e9c1c4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48478/0 (socket says 172.21.15.103:48478) 2023-07-02T09:03:15.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0e9bfff700 1 -- 172.21.15.103:0/3562806275 learned_addr learned my addr 172.21.15.103:0/3562806275 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:15.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0e9bfff700 1 -- 172.21.15.103:0/3562806275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c128e00 con 0x7f0e9c097520 2023-07-02T09:03:15.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0e9bfff700 1 --2- 172.21.15.103:0/3562806275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c097520 0x7f0e9c1c4ec0 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:15.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0e98ff9700 1 -- 172.21.15.103:0/3562806275 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0e90004b80 con 0x7f0e9c097520 2023-07-02T09:03:15.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0e98ff9700 1 -- 172.21.15.103:0/3562806275 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0e90004ce0 con 0x7f0e9c097520 2023-07-02T09:03:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9c1274d0 con 0x7f0e9c097520 2023-07-02T09:03:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0e98ff9700 1 -- 172.21.15.103:0/3562806275 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0e9000c550 con 0x7f0e9c097520 2023-07-02T09:03:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.753+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9c129930 con 0x7f0e9c097520 2023-07-02T09:03:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.754+0000 7f0e98ff9700 1 -- 172.21.15.103:0/3562806275 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f0e9000c700 con 0x7f0e9c097520 2023-07-02T09:03:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.754+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9c061ea0 con 0x7f0e9c097520 2023-07-02T09:03:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.754+0000 7f0e98ff9700 1 --2- 172.21.15.103:0/3562806275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0e88037770 0x7f0e88039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.754+0000 7f0e98ff9700 1 -- 172.21.15.103:0/3562806275 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f0e90046a20 con 0x7f0e9c097520 2023-07-02T09:03:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.755+0000 7f0e9b7fe700 1 --2- 172.21.15.103:0/3562806275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0e88037770 0x7f0e88039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:15.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.755+0000 7f0e9b7fe700 1 --2- 172.21.15.103:0/3562806275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0e88037770 0x7f0e88039c80 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:15.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.759+0000 7f0e98ff9700 1 -- 172.21.15.103:0/3562806275 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f0e90009a80 con 0x7f0e9c097520 2023-07-02T09:03:15.952 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.951+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0e9c073ca0 con 0x7f0e9c097520 2023-07-02T09:03:15.952 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.951+0000 7f0e98ff9700 1 -- 172.21.15.103:0/3562806275 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f0e900098a0 con 0x7f0e9c097520 2023-07-02T09:03:15.952 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:15.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0e88037770 msgr2=0x7f0e88039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:15.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 --2- 172.21.15.103:0/3562806275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0e88037770 0x7f0e88039c80 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:15.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c097520 msgr2=0x7f0e9c1c4ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:15.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 --2- 172.21.15.103:0/3562806275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c097520 0x7f0e9c1c4ec0 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:15.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 shutdown_connections 2023-07-02T09:03:15.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 --2- 172.21.15.103:0/3562806275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f0e88037770 0x7f0e88039c80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:15.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 --2- 172.21.15.103:0/3562806275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e9c097520 0x7f0e9c1c4ec0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:15.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 >> 172.21.15.103:0/3562806275 conn(0x7f0e9c0a1030 msgr2=0x7f0e9c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:15.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 shutdown_connections 2023-07-02T09:03:15.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:15.953+0000 7f0ea2700700 1 -- 172.21.15.103:0/3562806275 wait complete. 2023-07-02T09:03:15.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:16.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:17.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.107+0000 7f659c242700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:17.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.112+0000 7f659c242700 1 Processor -- start 2023-07-02T09:03:17.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.112+0000 7f659c242700 1 -- start start 2023-07-02T09:03:17.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f659c242700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6594132030 0x7f6594134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:17.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f659c242700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659406df60 con 0x7f659412f250 2023-07-02T09:03:17.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f659c242700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659409a710 con 0x7f6594132030 2023-07-02T09:03:17.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f65997dd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6594132030 0x7f6594134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:17.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f6599fde700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f659412f250 0x7f659412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55814/0 (socket says 172.21.15.103:55814) 2023-07-02T09:03:17.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f6599fde700 1 -- 172.21.15.103:0/4088908686 learned_addr learned my addr 172.21.15.103:0/4088908686 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:17.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f65997dd700 1 --2- 172.21.15.103:0/4088908686 >> v2:172.21.15.103:3300/0 conn(0x7f6594132030 0x7f6594134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48494/0 (socket says 172.21.15.103:48494) 2023-07-02T09:03:17.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f65997dd700 1 -- 172.21.15.103:0/4088908686 >> v1:172.21.15.103:6789/0 conn(0x7f659412f250 legacy=0x7f659412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:17.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f6598fdc700 1 -- 172.21.15.103:0/4088908686 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143536724 0 0) 0x7f659406df60 con 0x7f659412f250 2023-07-02T09:03:17.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f65997dd700 1 -- 172.21.15.103:0/4088908686 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659412ed20 con 0x7f6594132030 2023-07-02T09:03:17.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f65997dd700 1 --2- 172.21.15.103:0/4088908686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6594132030 0x7f6594134450 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1c180a94700880 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:17.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f6598fdc700 1 -- 172.21.15.103:0/4088908686 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6584009cd0 con 0x7f6594132030 2023-07-02T09:03:17.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f6598fdc700 1 -- 172.21.15.103:0/4088908686 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6584009e30 con 0x7f6594132030 2023-07-02T09:03:17.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.113+0000 7f6598fdc700 1 -- 172.21.15.103:0/4088908686 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f658400e360 con 0x7f6594132030 2023-07-02T09:03:17.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.114+0000 7f659c242700 1 -- 172.21.15.103:0/4088908686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6594132030 msgr2=0x7f6594134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:17.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.114+0000 7f659c242700 1 --2- 172.21.15.103:0/4088908686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6594132030 0x7f6594134450 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:17.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.114+0000 7f659c242700 1 -- 172.21.15.103:0/4088908686 shutdown_connections 2023-07-02T09:03:17.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.114+0000 7f659c242700 1 --2- 172.21.15.103:0/4088908686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6594132030 0x7f6594134450 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:17.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.114+0000 7f659c242700 1 -- 172.21.15.103:0/4088908686 >> 172.21.15.103:0/4088908686 conn(0x7f659409d810 msgr2=0x7f659409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:17.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.114+0000 7f659c242700 1 -- 172.21.15.103:0/4088908686 shutdown_connections 2023-07-02T09:03:17.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.114+0000 7f659c242700 1 -- 172.21.15.103:0/4088908686 wait complete. 2023-07-02T09:03:17.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.114+0000 7f659c242700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:17.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.115+0000 7f659c242700 1 Processor -- start 2023-07-02T09:03:17.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.115+0000 7f659c242700 1 -- start start 2023-07-02T09:03:17.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.116+0000 7f659c242700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659412f250 0x7f6594070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:17.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.116+0000 7f659c242700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65941304d0 con 0x7f659412f250 2023-07-02T09:03:17.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.116+0000 7f6599fde700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659412f250 0x7f6594070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:17.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.116+0000 7f6599fde700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659412f250 0x7f6594070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48506/0 (socket says 172.21.15.103:48506) 2023-07-02T09:03:17.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.116+0000 7f6599fde700 1 -- 172.21.15.103:0/2289770275 learned_addr learned my addr 172.21.15.103:0/2289770275 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:17.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.116+0000 7f6599fde700 1 -- 172.21.15.103:0/2289770275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594118590 con 0x7f659412f250 2023-07-02T09:03:17.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.116+0000 7f6599fde700 1 --2- 172.21.15.103:0/2289770275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659412f250 0x7f6594070ef0 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:17.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.117+0000 7f658a7fc700 1 -- 172.21.15.103:0/2289770275 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6590004b80 con 0x7f659412f250 2023-07-02T09:03:17.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.117+0000 7f658a7fc700 1 -- 172.21.15.103:0/2289770275 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6590004ce0 con 0x7f659412f250 2023-07-02T09:03:17.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.117+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65941308f0 con 0x7f659412f250 2023-07-02T09:03:17.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.117+0000 7f658a7fc700 1 -- 172.21.15.103:0/2289770275 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f659000c4f0 con 0x7f659412f250 2023-07-02T09:03:17.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.117+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6594117a60 con 0x7f659412f250 2023-07-02T09:03:17.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.117+0000 7f658a7fc700 1 -- 172.21.15.103:0/2289770275 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f659000c650 con 0x7f659412f250 2023-07-02T09:03:17.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.118+0000 7f658a7fc700 1 --2- 172.21.15.103:0/2289770275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65780377c0 0x7f6578039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:17.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.118+0000 7f658a7fc700 1 -- 172.21.15.103:0/2289770275 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f65900468e0 con 0x7f659412f250 2023-07-02T09:03:17.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.118+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6594061ea0 con 0x7f659412f250 2023-07-02T09:03:17.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.118+0000 7f65997dd700 1 --2- 172.21.15.103:0/2289770275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65780377c0 0x7f6578039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:17.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.119+0000 7f65997dd700 1 --2- 172.21.15.103:0/2289770275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65780377c0 0x7f6578039cd0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:17.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.123+0000 7f658a7fc700 1 -- 172.21.15.103:0/2289770275 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6590009c80 con 0x7f659412f250 2023-07-02T09:03:17.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.310+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6594073ca0 con 0x7f659412f250 2023-07-02T09:03:17.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.310+0000 7f658a7fc700 1 -- 172.21.15.103:0/2289770275 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f659002c020 con 0x7f659412f250 2023-07-02T09:03:17.312 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:17.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65780377c0 msgr2=0x7f6578039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:17.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 --2- 172.21.15.103:0/2289770275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65780377c0 0x7f6578039cd0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:17.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659412f250 msgr2=0x7f6594070ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:17.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 --2- 172.21.15.103:0/2289770275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659412f250 0x7f6594070ef0 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:17.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 shutdown_connections 2023-07-02T09:03:17.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 --2- 172.21.15.103:0/2289770275 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f65780377c0 0x7f6578039cd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:17.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 --2- 172.21.15.103:0/2289770275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f659412f250 0x7f6594070ef0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:17.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 >> 172.21.15.103:0/2289770275 conn(0x7f659409d810 msgr2=0x7f6594128990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:17.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 shutdown_connections 2023-07-02T09:03:17.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:17.312+0000 7f659c242700 1 -- 172.21.15.103:0/2289770275 wait complete. 2023-07-02T09:03:17.326 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:18.327 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:18.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.466+0000 7f29f930e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:18.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.471+0000 7f29f930e700 1 Processor -- start 2023-07-02T09:03:18.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.471+0000 7f29f930e700 1 -- start start 2023-07-02T09:03:18.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f930e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f29f4132030 0x7f29f4134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:18.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f930e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f406df60 con 0x7f29f412f250 2023-07-02T09:03:18.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f930e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f409a710 con 0x7f29f4132030 2023-07-02T09:03:18.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f29f4132030 0x7f29f4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f2ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f29f412f250 0x7f29f412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55830/0 (socket says 172.21.15.103:55830) 2023-07-02T09:03:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f2ffd700 1 -- 172.21.15.103:0/2621228330 learned_addr learned my addr 172.21.15.103:0/2621228330 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f27fc700 1 --2- 172.21.15.103:0/2621228330 >> v2:172.21.15.103:3300/0 conn(0x7f29f4132030 0x7f29f4134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48510/0 (socket says 172.21.15.103:48510) 2023-07-02T09:03:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f27fc700 1 -- 172.21.15.103:0/2621228330 >> v1:172.21.15.103:6789/0 conn(0x7f29f412f250 legacy=0x7f29f412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:18.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f27fc700 1 -- 172.21.15.103:0/2621228330 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f412ed20 con 0x7f29f4132030 2023-07-02T09:03:18.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f1ffb700 1 -- 172.21.15.103:0/2621228330 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890979045 0 0) 0x7f29f406df60 con 0x7f29f412f250 2023-07-02T09:03:18.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f27fc700 1 --2- 172.21.15.103:0/2621228330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f4132030 0x7f29f4134450 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247eaec1e00c312a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:18.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f1ffb700 1 -- 172.21.15.103:0/2621228330 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f29e4009cd0 con 0x7f29f4132030 2023-07-02T09:03:18.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f1ffb700 1 -- 172.21.15.103:0/2621228330 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f29e4009e30 con 0x7f29f4132030 2023-07-02T09:03:18.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.472+0000 7f29f1ffb700 1 -- 172.21.15.103:0/2621228330 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f29e400e360 con 0x7f29f4132030 2023-07-02T09:03:18.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.473+0000 7f29f930e700 1 -- 172.21.15.103:0/2621228330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f4132030 msgr2=0x7f29f4134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:18.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.473+0000 7f29f930e700 1 --2- 172.21.15.103:0/2621228330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f4132030 0x7f29f4134450 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:18.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.473+0000 7f29f930e700 1 -- 172.21.15.103:0/2621228330 shutdown_connections 2023-07-02T09:03:18.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.473+0000 7f29f930e700 1 --2- 172.21.15.103:0/2621228330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f4132030 0x7f29f4134450 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:18.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.473+0000 7f29f930e700 1 -- 172.21.15.103:0/2621228330 >> 172.21.15.103:0/2621228330 conn(0x7f29f409d810 msgr2=0x7f29f409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:18.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.473+0000 7f29f930e700 1 -- 172.21.15.103:0/2621228330 shutdown_connections 2023-07-02T09:03:18.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.473+0000 7f29f930e700 1 -- 172.21.15.103:0/2621228330 wait complete. 2023-07-02T09:03:18.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.473+0000 7f29f930e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:18.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.474+0000 7f29f930e700 1 Processor -- start 2023-07-02T09:03:18.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.474+0000 7f29f930e700 1 -- start start 2023-07-02T09:03:18.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.475+0000 7f29f930e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f412f250 0x7f29f4070530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:18.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.475+0000 7f29f930e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f4129360 con 0x7f29f412f250 2023-07-02T09:03:18.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.475+0000 7f29f2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f412f250 0x7f29f4070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:18.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.475+0000 7f29f2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f412f250 0x7f29f4070530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48516/0 (socket says 172.21.15.103:48516) 2023-07-02T09:03:18.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.475+0000 7f29f2ffd700 1 -- 172.21.15.103:0/3274354953 learned_addr learned my addr 172.21.15.103:0/3274354953 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:18.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.476+0000 7f29f2ffd700 1 -- 172.21.15.103:0/3274354953 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f4071ef0 con 0x7f29f412f250 2023-07-02T09:03:18.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.476+0000 7f29f2ffd700 1 --2- 172.21.15.103:0/3274354953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f412f250 0x7f29f4070530 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:18.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.476+0000 7f29db7fe700 1 -- 172.21.15.103:0/3274354953 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f29e8004b80 con 0x7f29f412f250 2023-07-02T09:03:18.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.476+0000 7f29db7fe700 1 -- 172.21.15.103:0/3274354953 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f29e8004ce0 con 0x7f29f412f250 2023-07-02T09:03:18.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.476+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f4129070 con 0x7f29f412f250 2023-07-02T09:03:18.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.476+0000 7f29db7fe700 1 -- 172.21.15.103:0/3274354953 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f29e800c4f0 con 0x7f29f412f250 2023-07-02T09:03:18.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.476+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f409cbf0 con 0x7f29f412f250 2023-07-02T09:03:18.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.477+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f4071a00 con 0x7f29f412f250 2023-07-02T09:03:18.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.477+0000 7f29db7fe700 1 -- 172.21.15.103:0/3274354953 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f29e800c650 con 0x7f29f412f250 2023-07-02T09:03:18.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.477+0000 7f29db7fe700 1 --2- 172.21.15.103:0/3274354953 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f29e00377c0 0x7f29e0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:18.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.477+0000 7f29db7fe700 1 -- 172.21.15.103:0/3274354953 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f29e8046990 con 0x7f29f412f250 2023-07-02T09:03:18.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.477+0000 7f29f27fc700 1 --2- 172.21.15.103:0/3274354953 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f29e00377c0 0x7f29e0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.478+0000 7f29f27fc700 1 --2- 172.21.15.103:0/3274354953 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f29e00377c0 0x7f29e0039cd0 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.482+0000 7f29db7fe700 1 -- 172.21.15.103:0/3274354953 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f29e8009c80 con 0x7f29f412f250 2023-07-02T09:03:18.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.673+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f29f4061ea0 con 0x7f29f412f250 2023-07-02T09:03:18.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.673+0000 7f29db7fe700 1 -- 172.21.15.103:0/3274354953 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f29e80098a0 con 0x7f29f412f250 2023-07-02T09:03:18.674 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f29e00377c0 msgr2=0x7f29e0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 --2- 172.21.15.103:0/3274354953 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f29e00377c0 0x7f29e0039cd0 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f412f250 msgr2=0x7f29f4070530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 --2- 172.21.15.103:0/3274354953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f412f250 0x7f29f4070530 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 shutdown_connections 2023-07-02T09:03:18.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 --2- 172.21.15.103:0/3274354953 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f29e00377c0 0x7f29e0039cd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:18.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 --2- 172.21.15.103:0/3274354953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29f412f250 0x7f29f4070530 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:18.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 >> 172.21.15.103:0/3274354953 conn(0x7f29f409d810 msgr2=0x7f29f412d660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:18.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 shutdown_connections 2023-07-02T09:03:18.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:18.675+0000 7f29f930e700 1 -- 172.21.15.103:0/3274354953 wait complete. 2023-07-02T09:03:18.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:19.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:19.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.825+0000 7f9eb91ae700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:19.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.830+0000 7f9eb91ae700 1 Processor -- start 2023-07-02T09:03:19.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.830+0000 7f9eb91ae700 1 -- start start 2023-07-02T09:03:19.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb91ae700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9eb4132030 0x7f9eb4134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:19.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb91ae700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb406df60 con 0x7f9eb412f250 2023-07-02T09:03:19.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb91ae700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb409a710 con 0x7f9eb4132030 2023-07-02T09:03:19.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9eb4132030 0x7f9eb4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:19.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb2d9d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9eb412f250 0x7f9eb412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55844/0 (socket says 172.21.15.103:55844) 2023-07-02T09:03:19.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb2d9d700 1 -- 172.21.15.103:0/3091928290 learned_addr learned my addr 172.21.15.103:0/3091928290 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:19.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb259c700 1 -- 172.21.15.103:0/3091928290 >> v1:172.21.15.103:6789/0 conn(0x7f9eb412f250 legacy=0x7f9eb412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:19.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb259c700 1 -- 172.21.15.103:0/3091928290 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb412ed20 con 0x7f9eb4132030 2023-07-02T09:03:19.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb1d9b700 1 -- 172.21.15.103:0/3091928290 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242460 0 0) 0x7f9eb406df60 con 0x7f9eb412f250 2023-07-02T09:03:19.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb259c700 1 --2- 172.21.15.103:0/3091928290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb4132030 0x7f9eb4134450 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7913fc1e3a08428 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:19.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.831+0000 7f9eb1d9b700 1 -- 172.21.15.103:0/3091928290 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9ea4009cd0 con 0x7f9eb4132030 2023-07-02T09:03:19.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb1d9b700 1 -- 172.21.15.103:0/3091928290 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9ea4009e30 con 0x7f9eb4132030 2023-07-02T09:03:19.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb1d9b700 1 -- 172.21.15.103:0/3091928290 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9ea400e360 con 0x7f9eb4132030 2023-07-02T09:03:19.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb91ae700 1 -- 172.21.15.103:0/3091928290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb4132030 msgr2=0x7f9eb4134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:19.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb91ae700 1 --2- 172.21.15.103:0/3091928290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb4132030 0x7f9eb4134450 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:19.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb91ae700 1 -- 172.21.15.103:0/3091928290 shutdown_connections 2023-07-02T09:03:19.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb91ae700 1 --2- 172.21.15.103:0/3091928290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb4132030 0x7f9eb4134450 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:19.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb91ae700 1 -- 172.21.15.103:0/3091928290 >> 172.21.15.103:0/3091928290 conn(0x7f9eb409d810 msgr2=0x7f9eb409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:19.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb91ae700 1 -- 172.21.15.103:0/3091928290 shutdown_connections 2023-07-02T09:03:19.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb91ae700 1 -- 172.21.15.103:0/3091928290 wait complete. 2023-07-02T09:03:19.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.832+0000 7f9eb91ae700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:19.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.834+0000 7f9eb91ae700 1 Processor -- start 2023-07-02T09:03:19.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.834+0000 7f9eb91ae700 1 -- start start 2023-07-02T09:03:19.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.834+0000 7f9eb91ae700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb412f250 0x7f9eb406ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:19.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.834+0000 7f9eb91ae700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb411c560 con 0x7f9eb412f250 2023-07-02T09:03:19.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.834+0000 7f9eb2d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb412f250 0x7f9eb406ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:19.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.834+0000 7f9eb2d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb412f250 0x7f9eb406ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48542/0 (socket says 172.21.15.103:48542) 2023-07-02T09:03:19.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.834+0000 7f9eb2d9d700 1 -- 172.21.15.103:0/432131628 learned_addr learned my addr 172.21.15.103:0/432131628 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:19.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.835+0000 7f9eb2d9d700 1 -- 172.21.15.103:0/432131628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb411c7c0 con 0x7f9eb412f250 2023-07-02T09:03:19.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.835+0000 7f9eb2d9d700 1 --2- 172.21.15.103:0/432131628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb412f250 0x7f9eb406ff70 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:19.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.835+0000 7f9e9b7fe700 1 -- 172.21.15.103:0/432131628 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9ea8004b80 con 0x7f9eb412f250 2023-07-02T09:03:19.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.835+0000 7f9e9b7fe700 1 -- 172.21.15.103:0/432131628 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9ea8004ce0 con 0x7f9eb412f250 2023-07-02T09:03:19.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.835+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb4071bd0 con 0x7f9eb412f250 2023-07-02T09:03:19.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.835+0000 7f9e9b7fe700 1 -- 172.21.15.103:0/432131628 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9ea800c4f0 con 0x7f9eb412f250 2023-07-02T09:03:19.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.835+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb411d420 con 0x7f9eb412f250 2023-07-02T09:03:19.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.836+0000 7f9e9b7fe700 1 -- 172.21.15.103:0/432131628 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f9ea800c650 con 0x7f9eb412f250 2023-07-02T09:03:19.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.836+0000 7f9e9b7fe700 1 --2- 172.21.15.103:0/432131628 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9ea00377c0 0x7f9ea0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:19.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.836+0000 7f9e9b7fe700 1 -- 172.21.15.103:0/432131628 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9ea8046990 con 0x7f9eb412f250 2023-07-02T09:03:19.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.836+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb411b770 con 0x7f9eb412f250 2023-07-02T09:03:19.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.836+0000 7f9eb259c700 1 --2- 172.21.15.103:0/432131628 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9ea00377c0 0x7f9ea0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:19.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.837+0000 7f9eb259c700 1 --2- 172.21.15.103:0/432131628 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9ea00377c0 0x7f9ea0039cd0 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:19.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:19.841+0000 7f9e9b7fe700 1 -- 172.21.15.103:0/432131628 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f9ea800e1c0 con 0x7f9eb412f250 2023-07-02T09:03:20.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.030+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9eb4061ea0 con 0x7f9eb412f250 2023-07-02T09:03:20.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.030+0000 7f9e9b7fe700 1 -- 172.21.15.103:0/432131628 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f9ea80098a0 con 0x7f9eb412f250 2023-07-02T09:03:20.032 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:20.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9ea00377c0 msgr2=0x7f9ea0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:20.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 --2- 172.21.15.103:0/432131628 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9ea00377c0 0x7f9ea0039cd0 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:20.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb412f250 msgr2=0x7f9eb406ff70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:20.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 --2- 172.21.15.103:0/432131628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb412f250 0x7f9eb406ff70 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:20.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 shutdown_connections 2023-07-02T09:03:20.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 --2- 172.21.15.103:0/432131628 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f9ea00377c0 0x7f9ea0039cd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:20.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 --2- 172.21.15.103:0/432131628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb412f250 0x7f9eb406ff70 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:20.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 >> 172.21.15.103:0/432131628 conn(0x7f9eb409d810 msgr2=0x7f9eb412e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:20.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.032+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 shutdown_connections 2023-07-02T09:03:20.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:20.033+0000 7f9eb91ae700 1 -- 172.21.15.103:0/432131628 wait complete. 2023-07-02T09:03:20.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:20.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2023-07-02T09:03:21.048 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:21.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.187+0000 7ff8d7b40700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:21.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.192+0000 7ff8d7b40700 1 Processor -- start 2023-07-02T09:03:21.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.192+0000 7ff8d7b40700 1 -- start start 2023-07-02T09:03:21.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d7b40700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff8d0133360 0x7ff8d009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:21.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d7b40700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d009e260 con 0x7ff8d0132f50 2023-07-02T09:03:21.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d7b40700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d00a1640 con 0x7ff8d0133360 2023-07-02T09:03:21.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d50db700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff8d0133360 0x7ff8d009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:21.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d58dc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff8d0132f50 0x7ff8d009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55854/0 (socket says 172.21.15.103:55854) 2023-07-02T09:03:21.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d58dc700 1 -- 172.21.15.103:0/83759294 learned_addr learned my addr 172.21.15.103:0/83759294 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:21.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d50db700 1 -- 172.21.15.103:0/83759294 >> v1:172.21.15.103:6789/0 conn(0x7ff8d0132f50 legacy=0x7ff8d009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:21.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d50db700 1 -- 172.21.15.103:0/83759294 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d0132a20 con 0x7ff8d0133360 2023-07-02T09:03:21.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d50db700 1 --2- 172.21.15.103:0/83759294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0133360 0x7ff8d009cc90 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f01aaf61135196 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:21.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.193+0000 7ff8d48da700 1 -- 172.21.15.103:0/83759294 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff8cc009cd0 con 0x7ff8d0133360 2023-07-02T09:03:21.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d48da700 1 -- 172.21.15.103:0/83759294 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff8cc009e30 con 0x7ff8d0133360 2023-07-02T09:03:21.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d48da700 1 -- 172.21.15.103:0/83759294 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff8cc00e360 con 0x7ff8d0133360 2023-07-02T09:03:21.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d7b40700 1 -- 172.21.15.103:0/83759294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0133360 msgr2=0x7ff8d009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:21.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d7b40700 1 --2- 172.21.15.103:0/83759294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0133360 0x7ff8d009cc90 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:21.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d7b40700 1 -- 172.21.15.103:0/83759294 shutdown_connections 2023-07-02T09:03:21.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d7b40700 1 --2- 172.21.15.103:0/83759294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0133360 0x7ff8d009cc90 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:21.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d7b40700 1 -- 172.21.15.103:0/83759294 >> 172.21.15.103:0/83759294 conn(0x7ff8d00a2080 msgr2=0x7ff8d00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:21.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d7b40700 1 -- 172.21.15.103:0/83759294 shutdown_connections 2023-07-02T09:03:21.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.194+0000 7ff8d7b40700 1 -- 172.21.15.103:0/83759294 wait complete. 2023-07-02T09:03:21.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.195+0000 7ff8d7b40700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:21.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.196+0000 7ff8d7b40700 1 Processor -- start 2023-07-02T09:03:21.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.196+0000 7ff8d7b40700 1 -- start start 2023-07-02T09:03:21.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.196+0000 7ff8d7b40700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0132f50 0x7ff8d012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:21.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.196+0000 7ff8d7b40700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d006d260 con 0x7ff8d0132f50 2023-07-02T09:03:21.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.196+0000 7ff8d58dc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0132f50 0x7ff8d012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:21.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.196+0000 7ff8d58dc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0132f50 0x7ff8d012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48548/0 (socket says 172.21.15.103:48548) 2023-07-02T09:03:21.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.196+0000 7ff8d58dc700 1 -- 172.21.15.103:0/1088407697 learned_addr learned my addr 172.21.15.103:0/1088407697 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:21.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.197+0000 7ff8d58dc700 1 -- 172.21.15.103:0/1088407697 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d012bb90 con 0x7ff8d0132f50 2023-07-02T09:03:21.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.197+0000 7ff8d58dc700 1 --2- 172.21.15.103:0/1088407697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0132f50 0x7ff8d012bea0 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:21.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.197+0000 7ff8c67fc700 1 -- 172.21.15.103:0/1088407697 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff8c0005a60 con 0x7ff8d0132f50 2023-07-02T09:03:21.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.197+0000 7ff8c67fc700 1 -- 172.21.15.103:0/1088407697 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff8c0005bc0 con 0x7ff8d0132f50 2023-07-02T09:03:21.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.197+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d006f730 con 0x7ff8d0132f50 2023-07-02T09:03:21.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.197+0000 7ff8c67fc700 1 -- 172.21.15.103:0/1088407697 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff8c0005e60 con 0x7ff8d0132f50 2023-07-02T09:03:21.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.197+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d006f490 con 0x7ff8d0132f50 2023-07-02T09:03:21.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.198+0000 7ff8c67fc700 1 -- 172.21.15.103:0/1088407697 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ff8c00070b0 con 0x7ff8d0132f50 2023-07-02T09:03:21.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.198+0000 7ff8c67fc700 1 --2- 172.21.15.103:0/1088407697 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8b40377c0 0x7ff8b4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:21.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.198+0000 7ff8c67fc700 1 -- 172.21.15.103:0/1088407697 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ff8c00469f0 con 0x7ff8d0132f50 2023-07-02T09:03:21.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.198+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d006f490 con 0x7ff8d0132f50 2023-07-02T09:03:21.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.198+0000 7ff8d50db700 1 --2- 172.21.15.103:0/1088407697 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8b40377c0 0x7ff8b4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:21.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.199+0000 7ff8d50db700 1 --2- 172.21.15.103:0/1088407697 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8b40377c0 0x7ff8b4039cd0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:21.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.203+0000 7ff8c67fc700 1 -- 172.21.15.103:0/1088407697 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ff8d006f490 con 0x7ff8d0132f50 2023-07-02T09:03:21.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.393+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff8d006f490 con 0x7ff8d0132f50 2023-07-02T09:03:21.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.393+0000 7ff8c67fc700 1 -- 172.21.15.103:0/1088407697 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ff8d006f490 con 0x7ff8d0132f50 2023-07-02T09:03:21.395 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:21.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8b40377c0 msgr2=0x7ff8b4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:21.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 --2- 172.21.15.103:0/1088407697 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8b40377c0 0x7ff8b4039cd0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:21.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0132f50 msgr2=0x7ff8d012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:21.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 --2- 172.21.15.103:0/1088407697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0132f50 0x7ff8d012bea0 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:21.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 shutdown_connections 2023-07-02T09:03:21.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 --2- 172.21.15.103:0/1088407697 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8b40377c0 0x7ff8b4039cd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:21.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 --2- 172.21.15.103:0/1088407697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d0132f50 0x7ff8d012bea0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:21.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 >> 172.21.15.103:0/1088407697 conn(0x7ff8d00a2080 msgr2=0x7ff8d01bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:21.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 shutdown_connections 2023-07-02T09:03:21.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:21.395+0000 7ff8d7b40700 1 -- 172.21.15.103:0/1088407697 wait complete. 2023-07-02T09:03:21.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:22.410 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.549+0000 7f22aa8d4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:22.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.554+0000 7f22aa8d4700 1 Processor -- start 2023-07-02T09:03:22.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.554+0000 7f22aa8d4700 1 -- start start 2023-07-02T09:03:22.556 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22aa8d4700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f22a4132030 0x7f22a4134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:22.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22aa8d4700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a406df60 con 0x7f22a412f250 2023-07-02T09:03:22.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22aa8d4700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a409a710 con 0x7f22a4132030 2023-07-02T09:03:22.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a37fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f22a4132030 0x7f22a4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:22.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a37fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f22a4132030 0x7f22a4134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48562/0 (socket says 172.21.15.103:48562) 2023-07-02T09:03:22.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a37fe700 1 -- 172.21.15.103:0/2162519481 learned_addr learned my addr 172.21.15.103:0/2162519481 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:22.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a3fff700 1 --1- 172.21.15.103:0/2162519481 >> v1:172.21.15.103:6789/0 conn(0x7f22a412f250 0x7f22a412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55862/0 (socket says 172.21.15.103:55862) 2023-07-02T09:03:22.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a37fe700 1 -- 172.21.15.103:0/2162519481 >> v1:172.21.15.103:6789/0 conn(0x7f22a412f250 legacy=0x7f22a412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:22.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a37fe700 1 -- 172.21.15.103:0/2162519481 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a412ed20 con 0x7f22a4132030 2023-07-02T09:03:22.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a2ffd700 1 -- 172.21.15.103:0/2162519481 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689415341 0 0) 0x7f22a406df60 con 0x7f22a412f250 2023-07-02T09:03:22.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a37fe700 1 --2- 172.21.15.103:0/2162519481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a4132030 0x7f22a4134450 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b607e6a4a956805 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:22.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a2ffd700 1 -- 172.21.15.103:0/2162519481 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f229400d580 con 0x7f22a4132030 2023-07-02T09:03:22.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a2ffd700 1 -- 172.21.15.103:0/2162519481 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f229400d6e0 con 0x7f22a4132030 2023-07-02T09:03:22.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.555+0000 7f22a2ffd700 1 -- 172.21.15.103:0/2162519481 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f229400d980 con 0x7f22a4132030 2023-07-02T09:03:22.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.556+0000 7f22aa8d4700 1 -- 172.21.15.103:0/2162519481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a4132030 msgr2=0x7f22a4134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:22.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.556+0000 7f22aa8d4700 1 --2- 172.21.15.103:0/2162519481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a4132030 0x7f22a4134450 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:22.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.556+0000 7f22aa8d4700 1 -- 172.21.15.103:0/2162519481 shutdown_connections 2023-07-02T09:03:22.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.556+0000 7f22aa8d4700 1 --2- 172.21.15.103:0/2162519481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a4132030 0x7f22a4134450 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:22.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.556+0000 7f22aa8d4700 1 -- 172.21.15.103:0/2162519481 >> 172.21.15.103:0/2162519481 conn(0x7f22a409d810 msgr2=0x7f22a409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:22.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.556+0000 7f22aa8d4700 1 -- 172.21.15.103:0/2162519481 shutdown_connections 2023-07-02T09:03:22.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.556+0000 7f22aa8d4700 1 -- 172.21.15.103:0/2162519481 wait complete. 2023-07-02T09:03:22.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.556+0000 7f22aa8d4700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:22.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.557+0000 7f22aa8d4700 1 Processor -- start 2023-07-02T09:03:22.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.557+0000 7f22aa8d4700 1 -- start start 2023-07-02T09:03:22.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.558+0000 7f22aa8d4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a412f250 0x7f22a406fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:22.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.558+0000 7f22aa8d4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4071390 con 0x7f22a412f250 2023-07-02T09:03:22.562 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.558+0000 7f22a3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a412f250 0x7f22a406fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:22.562 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.558+0000 7f22a3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a412f250 0x7f22a406fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48576/0 (socket says 172.21.15.103:48576) 2023-07-02T09:03:22.562 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.558+0000 7f22a3fff700 1 -- 172.21.15.103:0/4090534976 learned_addr learned my addr 172.21.15.103:0/4090534976 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:22.562 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.558+0000 7f22a3fff700 1 -- 172.21.15.103:0/4090534976 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a41292e0 con 0x7f22a412f250 2023-07-02T09:03:22.562 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.559+0000 7f22a3fff700 1 --2- 172.21.15.103:0/4090534976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a412f250 0x7f22a406fe70 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:22.562 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.559+0000 7f22a0ff9700 1 -- 172.21.15.103:0/4090534976 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2298004b80 con 0x7f22a412f250 2023-07-02T09:03:22.562 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.559+0000 7f22a0ff9700 1 -- 172.21.15.103:0/4090534976 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2298004ce0 con 0x7f22a412f250 2023-07-02T09:03:22.562 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.559+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a4128f40 con 0x7f22a412f250 2023-07-02T09:03:22.563 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.559+0000 7f22a0ff9700 1 -- 172.21.15.103:0/4090534976 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f229800c4f0 con 0x7f22a412f250 2023-07-02T09:03:22.563 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.559+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a41179c0 con 0x7f22a412f250 2023-07-02T09:03:22.563 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.559+0000 7f22a0ff9700 1 -- 172.21.15.103:0/4090534976 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f229800c650 con 0x7f22a412f250 2023-07-02T09:03:22.564 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.560+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a4061ea0 con 0x7f22a412f250 2023-07-02T09:03:22.564 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.560+0000 7f22a0ff9700 1 --2- 172.21.15.103:0/4090534976 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22900377c0 0x7f2290039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:22.564 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.560+0000 7f22a0ff9700 1 -- 172.21.15.103:0/4090534976 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f22980468b0 con 0x7f22a412f250 2023-07-02T09:03:22.564 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.560+0000 7f22a37fe700 1 --2- 172.21.15.103:0/4090534976 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22900377c0 0x7f2290039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:22.564 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.561+0000 7f22a37fe700 1 --2- 172.21.15.103:0/4090534976 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22900377c0 0x7f2290039cd0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:22.565 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.564+0000 7f22a0ff9700 1 -- 172.21.15.103:0/4090534976 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2298009ae0 con 0x7f22a412f250 2023-07-02T09:03:22.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.749+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f22a41292c0 con 0x7f22a412f250 2023-07-02T09:03:22.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.749+0000 7f22a0ff9700 1 -- 172.21.15.103:0/4090534976 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f22980090b0 con 0x7f22a412f250 2023-07-02T09:03:22.751 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:22.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.751+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22900377c0 msgr2=0x7f2290039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:22.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.751+0000 7f22aa8d4700 1 --2- 172.21.15.103:0/4090534976 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22900377c0 0x7f2290039cd0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:22.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.751+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a412f250 msgr2=0x7f22a406fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:22.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.751+0000 7f22aa8d4700 1 --2- 172.21.15.103:0/4090534976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a412f250 0x7f22a406fe70 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:22.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.751+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 shutdown_connections 2023-07-02T09:03:22.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.751+0000 7f22aa8d4700 1 --2- 172.21.15.103:0/4090534976 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f22900377c0 0x7f2290039cd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:22.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.751+0000 7f22aa8d4700 1 --2- 172.21.15.103:0/4090534976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22a412f250 0x7f22a406fe70 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:22.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.751+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 >> 172.21.15.103:0/4090534976 conn(0x7f22a409d810 msgr2=0x7f22a412dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:22.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.752+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 shutdown_connections 2023-07-02T09:03:22.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:22.752+0000 7f22aa8d4700 1 -- 172.21.15.103:0/4090534976 wait complete. 2023-07-02T09:03:22.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:23.766 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:23.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.906+0000 7f4084c7d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:23.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.911+0000 7f4084c7d700 1 Processor -- start 2023-07-02T09:03:23.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.911+0000 7f4084c7d700 1 -- start start 2023-07-02T09:03:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f4084c7d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4080132030 0x7f4080134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f4084c7d700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408006df60 con 0x7f408012f250 2023-07-02T09:03:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f4084c7d700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408009a710 con 0x7f4080132030 2023-07-02T09:03:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f407dd9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4080132030 0x7f4080134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f407dd9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4080132030 0x7f4080134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42612/0 (socket says 172.21.15.103:42612) 2023-07-02T09:03:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f407dd9b700 1 -- 172.21.15.103:0/2865106733 learned_addr learned my addr 172.21.15.103:0/2865106733 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f407e59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f408012f250 0x7f408012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36514/0 (socket says 172.21.15.103:36514) 2023-07-02T09:03:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f407dd9b700 1 -- 172.21.15.103:0/2865106733 >> v1:172.21.15.103:6789/0 conn(0x7f408012f250 legacy=0x7f408012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.912+0000 7f407dd9b700 1 -- 172.21.15.103:0/2865106733 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408012ed20 con 0x7f4080132030 2023-07-02T09:03:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f407dd9b700 1 --2- 172.21.15.103:0/2865106733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4080132030 0x7f4080134450 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60daf004881d5a65 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f407d59a700 1 -- 172.21.15.103:0/2865106733 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f407000d580 con 0x7f4080132030 2023-07-02T09:03:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f407d59a700 1 -- 172.21.15.103:0/2865106733 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f407000d6e0 con 0x7f4080132030 2023-07-02T09:03:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f407d59a700 1 -- 172.21.15.103:0/2865106733 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f407000d980 con 0x7f4080132030 2023-07-02T09:03:23.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f4084c7d700 1 -- 172.21.15.103:0/2865106733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4080132030 msgr2=0x7f4080134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:23.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f4084c7d700 1 --2- 172.21.15.103:0/2865106733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4080132030 0x7f4080134450 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:23.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f4084c7d700 1 -- 172.21.15.103:0/2865106733 shutdown_connections 2023-07-02T09:03:23.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f4084c7d700 1 --2- 172.21.15.103:0/2865106733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4080132030 0x7f4080134450 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:23.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f4084c7d700 1 -- 172.21.15.103:0/2865106733 >> 172.21.15.103:0/2865106733 conn(0x7f408009d810 msgr2=0x7f408009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:23.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.913+0000 7f4084c7d700 1 -- 172.21.15.103:0/2865106733 shutdown_connections 2023-07-02T09:03:23.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.914+0000 7f4084c7d700 1 -- 172.21.15.103:0/2865106733 wait complete. 2023-07-02T09:03:23.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.914+0000 7f4084c7d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:23.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.915+0000 7f4084c7d700 1 Processor -- start 2023-07-02T09:03:23.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.915+0000 7f4084c7d700 1 -- start start 2023-07-02T09:03:23.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.916+0000 7f4084c7d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408012f250 0x7f4080070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:23.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.916+0000 7f4084c7d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4080071240 con 0x7f408012f250 2023-07-02T09:03:23.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.916+0000 7f407e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408012f250 0x7f4080070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:23.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.916+0000 7f407e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408012f250 0x7f4080070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42618/0 (socket says 172.21.15.103:42618) 2023-07-02T09:03:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.916+0000 7f407e59c700 1 -- 172.21.15.103:0/1144929902 learned_addr learned my addr 172.21.15.103:0/1144929902 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.916+0000 7f407e59c700 1 -- 172.21.15.103:0/1144929902 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4080119220 con 0x7f408012f250 2023-07-02T09:03:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.917+0000 7f407e59c700 1 --2- 172.21.15.103:0/1144929902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408012f250 0x7f4080070100 crc :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.917+0000 7f406effd700 1 -- 172.21.15.103:0/1144929902 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f40740059a0 con 0x7f408012f250 2023-07-02T09:03:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.917+0000 7f406effd700 1 -- 172.21.15.103:0/1144929902 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4074005b00 con 0x7f408012f250 2023-07-02T09:03:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.917+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4080071d60 con 0x7f408012f250 2023-07-02T09:03:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.917+0000 7f406effd700 1 -- 172.21.15.103:0/1144929902 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4074005da0 con 0x7f408012f250 2023-07-02T09:03:23.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.917+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4080118d50 con 0x7f408012f250 2023-07-02T09:03:23.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.918+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4080061ea0 con 0x7f408012f250 2023-07-02T09:03:23.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.918+0000 7f406effd700 1 -- 172.21.15.103:0/1144929902 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f40740070b0 con 0x7f408012f250 2023-07-02T09:03:23.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.918+0000 7f406effd700 1 --2- 172.21.15.103:0/1144929902 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4060037770 0x7f4060039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:23.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.918+0000 7f406effd700 1 -- 172.21.15.103:0/1144929902 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4074046b90 con 0x7f408012f250 2023-07-02T09:03:23.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.918+0000 7f407dd9b700 1 --2- 172.21.15.103:0/1144929902 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4060037770 0x7f4060039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:23.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.919+0000 7f407dd9b700 1 --2- 172.21.15.103:0/1144929902 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4060037770 0x7f4060039c80 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:23.924 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:23.923+0000 7f406effd700 1 -- 172.21.15.103:0/1144929902 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f407400a4e0 con 0x7f408012f250 2023-07-02T09:03:24.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.109+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f408012c6c0 con 0x7f408012f250 2023-07-02T09:03:24.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.110+0000 7f406effd700 1 -- 172.21.15.103:0/1144929902 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f407400a120 con 0x7f408012f250 2023-07-02T09:03:24.111 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:24.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4060037770 msgr2=0x7f4060039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:24.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 --2- 172.21.15.103:0/1144929902 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4060037770 0x7f4060039c80 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:24.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408012f250 msgr2=0x7f4080070100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:24.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 --2- 172.21.15.103:0/1144929902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408012f250 0x7f4080070100 crc :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:24.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 shutdown_connections 2023-07-02T09:03:24.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 --2- 172.21.15.103:0/1144929902 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4060037770 0x7f4060039c80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:24.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 --2- 172.21.15.103:0/1144929902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f408012f250 0x7f4080070100 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:24.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 >> 172.21.15.103:0/1144929902 conn(0x7f408009d810 msgr2=0x7f408012e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:24.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 shutdown_connections 2023-07-02T09:03:24.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:24.112+0000 7f4084c7d700 1 -- 172.21.15.103:0/1144929902 wait complete. 2023-07-02T09:03:24.125 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:25.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:25.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.262+0000 7fbd5129e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:25.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.267+0000 7fbd5129e700 1 Processor -- start 2023-07-02T09:03:25.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.267+0000 7fbd5129e700 1 -- start start 2023-07-02T09:03:25.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.268+0000 7fbd5129e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbd4c132f50 0x7fbd4c133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:25.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.268+0000 7fbd5129e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4c09e260 con 0x7fbd4c09c580 2023-07-02T09:03:25.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.268+0000 7fbd5129e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4c0a1640 con 0x7fbd4c132f50 2023-07-02T09:03:25.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.268+0000 7fbd4affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbd4c132f50 0x7fbd4c133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:25.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.268+0000 7fbd4a7fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fbd4c09c580 0x7fbd4c09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36530/0 (socket says 172.21.15.103:36530) 2023-07-02T09:03:25.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.268+0000 7fbd4a7fc700 1 -- 172.21.15.103:0/2457170600 learned_addr learned my addr 172.21.15.103:0/2457170600 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:25.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.268+0000 7fbd4affd700 1 -- 172.21.15.103:0/2457170600 >> v1:172.21.15.103:6789/0 conn(0x7fbd4c09c580 legacy=0x7fbd4c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:25.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd49ffb700 1 -- 172.21.15.103:0/2457170600 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527271411 0 0) 0x7fbd4c09e260 con 0x7fbd4c09c580 2023-07-02T09:03:25.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd4affd700 1 -- 172.21.15.103:0/2457170600 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c132a20 con 0x7fbd4c132f50 2023-07-02T09:03:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd4affd700 1 --2- 172.21.15.103:0/2457170600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c132f50 0x7fbd4c133360 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a7034024f3137f server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd49ffb700 1 -- 172.21.15.103:0/2457170600 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fbd40009cd0 con 0x7fbd4c132f50 2023-07-02T09:03:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd49ffb700 1 -- 172.21.15.103:0/2457170600 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fbd40009e30 con 0x7fbd4c132f50 2023-07-02T09:03:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd49ffb700 1 -- 172.21.15.103:0/2457170600 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fbd4000e360 con 0x7fbd4c132f50 2023-07-02T09:03:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd5129e700 1 -- 172.21.15.103:0/2457170600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c132f50 msgr2=0x7fbd4c133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd5129e700 1 --2- 172.21.15.103:0/2457170600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c132f50 0x7fbd4c133360 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd5129e700 1 -- 172.21.15.103:0/2457170600 shutdown_connections 2023-07-02T09:03:25.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd5129e700 1 --2- 172.21.15.103:0/2457170600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c132f50 0x7fbd4c133360 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:25.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd5129e700 1 -- 172.21.15.103:0/2457170600 >> 172.21.15.103:0/2457170600 conn(0x7fbd4c0a2080 msgr2=0x7fbd4c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:25.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd5129e700 1 -- 172.21.15.103:0/2457170600 shutdown_connections 2023-07-02T09:03:25.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.269+0000 7fbd5129e700 1 -- 172.21.15.103:0/2457170600 wait complete. 2023-07-02T09:03:25.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.270+0000 7fbd5129e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.271+0000 7fbd5129e700 1 Processor -- start 2023-07-02T09:03:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.271+0000 7fbd5129e700 1 -- start start 2023-07-02T09:03:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.271+0000 7fbd5129e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c09c580 0x7fbd4c1baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.271+0000 7fbd5129e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c1ba610 con 0x7fbd4c09c580 2023-07-02T09:03:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.271+0000 7fbd4affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c09c580 0x7fbd4c1baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd4affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c09c580 0x7fbd4c1baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42636/0 (socket says 172.21.15.103:42636) 2023-07-02T09:03:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd4affd700 1 -- 172.21.15.103:0/3509733779 learned_addr learned my addr 172.21.15.103:0/3509733779 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd4affd700 1 -- 172.21.15.103:0/3509733779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c06dfc0 con 0x7fbd4c09c580 2023-07-02T09:03:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd4affd700 1 --2- 172.21.15.103:0/3509733779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c09c580 0x7fbd4c1baa60 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd337fe700 1 -- 172.21.15.103:0/3509733779 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fbd40013530 con 0x7fbd4c09c580 2023-07-02T09:03:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd337fe700 1 -- 172.21.15.103:0/3509733779 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fbd40013690 con 0x7fbd4c09c580 2023-07-02T09:03:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4c1ba270 con 0x7fbd4c09c580 2023-07-02T09:03:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd337fe700 1 -- 172.21.15.103:0/3509733779 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fbd40013930 con 0x7fbd4c09c580 2023-07-02T09:03:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.272+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4c06dba0 con 0x7fbd4c09c580 2023-07-02T09:03:25.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.273+0000 7fbd337fe700 1 -- 172.21.15.103:0/3509733779 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fbd40013b30 con 0x7fbd4c09c580 2023-07-02T09:03:25.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.273+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4c1ba460 con 0x7fbd4c09c580 2023-07-02T09:03:25.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.273+0000 7fbd337fe700 1 --2- 172.21.15.103:0/3509733779 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd38037760 0x7fbd38039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:25.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.273+0000 7fbd337fe700 1 -- 172.21.15.103:0/3509733779 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fbd40018040 con 0x7fbd4c09c580 2023-07-02T09:03:25.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.273+0000 7fbd4a7fc700 1 --2- 172.21.15.103:0/3509733779 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd38037760 0x7fbd38039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:25.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.274+0000 7fbd4a7fc700 1 --2- 172.21.15.103:0/3509733779 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd38037760 0x7fbd38039c70 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:25.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.278+0000 7fbd337fe700 1 -- 172.21.15.103:0/3509733779 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fbd40018e00 con 0x7fbd4c09c580 2023-07-02T09:03:25.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.463+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbd4c061ea0 con 0x7fbd4c09c580 2023-07-02T09:03:25.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.463+0000 7fbd337fe700 1 -- 172.21.15.103:0/3509733779 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fbd40010e60 con 0x7fbd4c09c580 2023-07-02T09:03:25.465 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:25.466 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd38037760 msgr2=0x7fbd38039c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 --2- 172.21.15.103:0/3509733779 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd38037760 0x7fbd38039c70 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c09c580 msgr2=0x7fbd4c1baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 --2- 172.21.15.103:0/3509733779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c09c580 0x7fbd4c1baa60 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 shutdown_connections 2023-07-02T09:03:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 --2- 172.21.15.103:0/3509733779 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fbd38037760 0x7fbd38039c70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 --2- 172.21.15.103:0/3509733779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd4c09c580 0x7fbd4c1baa60 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 >> 172.21.15.103:0/3509733779 conn(0x7fbd4c0a2080 msgr2=0x7fbd4c130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 shutdown_connections 2023-07-02T09:03:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:25.465+0000 7fbd5129e700 1 -- 172.21.15.103:0/3509733779 wait complete. 2023-07-02T09:03:25.479 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:26.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2023-07-02T09:03:26.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.618+0000 7f2a97897700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.623+0000 7f2a97897700 1 Processor -- start 2023-07-02T09:03:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.623+0000 7f2a97897700 1 -- start start 2023-07-02T09:03:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.624+0000 7f2a97897700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2a90132f50 0x7f2a90133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.624+0000 7f2a97897700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9009e260 con 0x7f2a9009c580 2023-07-02T09:03:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.624+0000 7f2a97897700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a900a1640 con 0x7f2a90132f50 2023-07-02T09:03:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.624+0000 7f2a95633700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2a90132f50 0x7f2a90133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.624+0000 7f2a94e32700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2a9009c580 0x7f2a9009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36534/0 (socket says 172.21.15.103:36534) 2023-07-02T09:03:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.624+0000 7f2a95633700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2a90132f50 0x7f2a90133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42640/0 (socket says 172.21.15.103:42640) 2023-07-02T09:03:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.624+0000 7f2a94e32700 1 -- 172.21.15.103:0/2786557755 learned_addr learned my addr 172.21.15.103:0/2786557755 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:26.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.625+0000 7f2a95633700 1 -- 172.21.15.103:0/2786557755 >> v1:172.21.15.103:6789/0 conn(0x7f2a9009c580 legacy=0x7f2a9009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:26.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.625+0000 7f2a95633700 1 -- 172.21.15.103:0/2786557755 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a90132a20 con 0x7f2a90132f50 2023-07-02T09:03:26.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.625+0000 7f2a95633700 1 --2- 172.21.15.103:0/2786557755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a90132f50 0x7f2a90133360 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f94746ab87cda63 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:26.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.625+0000 7f2a87fff700 1 -- 172.21.15.103:0/2786557755 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2a80009cd0 con 0x7f2a90132f50 2023-07-02T09:03:26.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.625+0000 7f2a87fff700 1 -- 172.21.15.103:0/2786557755 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2a80009e30 con 0x7f2a90132f50 2023-07-02T09:03:26.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.625+0000 7f2a87fff700 1 -- 172.21.15.103:0/2786557755 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2a8000e360 con 0x7f2a90132f50 2023-07-02T09:03:26.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.626+0000 7f2a97897700 1 -- 172.21.15.103:0/2786557755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a90132f50 msgr2=0x7f2a90133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:26.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.626+0000 7f2a97897700 1 --2- 172.21.15.103:0/2786557755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a90132f50 0x7f2a90133360 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:26.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.626+0000 7f2a97897700 1 -- 172.21.15.103:0/2786557755 shutdown_connections 2023-07-02T09:03:26.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.626+0000 7f2a97897700 1 --2- 172.21.15.103:0/2786557755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a90132f50 0x7f2a90133360 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:26.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.626+0000 7f2a97897700 1 -- 172.21.15.103:0/2786557755 >> 172.21.15.103:0/2786557755 conn(0x7f2a900a2080 msgr2=0x7f2a900a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:26.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.626+0000 7f2a97897700 1 -- 172.21.15.103:0/2786557755 shutdown_connections 2023-07-02T09:03:26.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.626+0000 7f2a97897700 1 -- 172.21.15.103:0/2786557755 wait complete. 2023-07-02T09:03:26.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.626+0000 7f2a97897700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:26.630 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.627+0000 7f2a97897700 1 Processor -- start 2023-07-02T09:03:26.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.627+0000 7f2a97897700 1 -- start start 2023-07-02T09:03:26.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.628+0000 7f2a97897700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a9009c580 0x7f2a9012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:26.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.628+0000 7f2a97897700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9006f9d0 con 0x7f2a9009c580 2023-07-02T09:03:26.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.628+0000 7f2a95633700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a9009c580 0x7f2a9012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:26.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.628+0000 7f2a95633700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a9009c580 0x7f2a9012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42652/0 (socket says 172.21.15.103:42652) 2023-07-02T09:03:26.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.628+0000 7f2a95633700 1 -- 172.21.15.103:0/1654921378 learned_addr learned my addr 172.21.15.103:0/1654921378 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:26.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.628+0000 7f2a95633700 1 -- 172.21.15.103:0/1654921378 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a90133a00 con 0x7f2a9009c580 2023-07-02T09:03:26.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.629+0000 7f2a95633700 1 --2- 172.21.15.103:0/1654921378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a9009c580 0x7f2a9012b7b0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:26.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.629+0000 7f2a85ffb700 1 -- 172.21.15.103:0/1654921378 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2a80013530 con 0x7f2a9009c580 2023-07-02T09:03:26.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.629+0000 7f2a85ffb700 1 -- 172.21.15.103:0/1654921378 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2a80013690 con 0x7f2a9009c580 2023-07-02T09:03:26.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.629+0000 7f2a85ffb700 1 -- 172.21.15.103:0/1654921378 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2a80013930 con 0x7f2a9009c580 2023-07-02T09:03:26.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.629+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9006d160 con 0x7f2a9009c580 2023-07-02T09:03:26.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.629+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9006da70 con 0x7f2a9009c580 2023-07-02T09:03:26.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.630+0000 7f2a85ffb700 1 -- 172.21.15.103:0/1654921378 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2a80013ae0 con 0x7f2a9009c580 2023-07-02T09:03:26.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.630+0000 7f2a85ffb700 1 --2- 172.21.15.103:0/1654921378 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2a740377c0 0x7f2a74039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:26.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.630+0000 7f2a85ffb700 1 -- 172.21.15.103:0/1654921378 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2a80018040 con 0x7f2a9009c580 2023-07-02T09:03:26.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.630+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a90061ea0 con 0x7f2a9009c580 2023-07-02T09:03:26.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.630+0000 7f2a94e32700 1 --2- 172.21.15.103:0/1654921378 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2a740377c0 0x7f2a74039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:26.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.631+0000 7f2a94e32700 1 --2- 172.21.15.103:0/1654921378 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2a740377c0 0x7f2a74039cd0 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.635+0000 7f2a85ffb700 1 -- 172.21.15.103:0/1654921378 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2a80018dd0 con 0x7f2a9009c580 2023-07-02T09:03:26.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.821+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2a900725b0 con 0x7f2a9009c580 2023-07-02T09:03:26.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.821+0000 7f2a85ffb700 1 -- 172.21.15.103:0/1654921378 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2a800189f0 con 0x7f2a9009c580 2023-07-02T09:03:26.823 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:26.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2a740377c0 msgr2=0x7f2a74039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:26.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 --2- 172.21.15.103:0/1654921378 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2a740377c0 0x7f2a74039cd0 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:26.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a9009c580 msgr2=0x7f2a9012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:26.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 --2- 172.21.15.103:0/1654921378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a9009c580 0x7f2a9012b7b0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:26.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 shutdown_connections 2023-07-02T09:03:26.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 --2- 172.21.15.103:0/1654921378 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2a740377c0 0x7f2a74039cd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:26.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 --2- 172.21.15.103:0/1654921378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a9009c580 0x7f2a9012b7b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:26.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 >> 172.21.15.103:0/1654921378 conn(0x7f2a900a2080 msgr2=0x7f2a901babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:26.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 shutdown_connections 2023-07-02T09:03:26.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:26.823+0000 7f2a97897700 1 -- 172.21.15.103:0/1654921378 wait complete. 2023-07-02T09:03:26.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:27.839 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:27.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.976+0000 7f2ea5b45700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:27.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.981+0000 7f2ea5b45700 1 Processor -- start 2023-07-02T09:03:27.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.981+0000 7f2ea5b45700 1 -- start start 2023-07-02T09:03:27.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.982+0000 7f2ea5b45700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2ea0132030 0x7f2ea0134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:27.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.982+0000 7f2ea5b45700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea006df60 con 0x7f2ea012f250 2023-07-02T09:03:27.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.982+0000 7f2ea5b45700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea009a710 con 0x7f2ea0132030 2023-07-02T09:03:27.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.982+0000 7f2e9effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2ea0132030 0x7f2ea0134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:27.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.982+0000 7f2e9f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2ea012f250 0x7f2ea012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36550/0 (socket says 172.21.15.103:36550) 2023-07-02T09:03:27.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.982+0000 7f2e9effd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2ea0132030 0x7f2ea0134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42660/0 (socket says 172.21.15.103:42660) 2023-07-02T09:03:27.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.982+0000 7f2e9f7fe700 1 -- 172.21.15.103:0/924562138 learned_addr learned my addr 172.21.15.103:0/924562138 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:27.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2e9effd700 1 -- 172.21.15.103:0/924562138 >> v1:172.21.15.103:6789/0 conn(0x7f2ea012f250 legacy=0x7f2ea012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:27.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2e9effd700 1 -- 172.21.15.103:0/924562138 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea012ed20 con 0x7f2ea0132030 2023-07-02T09:03:27.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2e9e7fc700 1 -- 172.21.15.103:0/924562138 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585410817 0 0) 0x7f2ea006df60 con 0x7f2ea012f250 2023-07-02T09:03:27.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2e9effd700 1 --2- 172.21.15.103:0/924562138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea0132030 0x7f2ea0134450 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340d1f9ec2fa39a7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:27.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2e9e7fc700 1 -- 172.21.15.103:0/924562138 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2e90009cd0 con 0x7f2ea0132030 2023-07-02T09:03:27.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2e9e7fc700 1 -- 172.21.15.103:0/924562138 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2e90009e30 con 0x7f2ea0132030 2023-07-02T09:03:27.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2e9e7fc700 1 -- 172.21.15.103:0/924562138 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2e9000e360 con 0x7f2ea0132030 2023-07-02T09:03:27.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2ea5b45700 1 -- 172.21.15.103:0/924562138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea0132030 msgr2=0x7f2ea0134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.983+0000 7f2ea5b45700 1 --2- 172.21.15.103:0/924562138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea0132030 0x7f2ea0134450 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.984+0000 7f2ea5b45700 1 -- 172.21.15.103:0/924562138 shutdown_connections 2023-07-02T09:03:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.984+0000 7f2ea5b45700 1 --2- 172.21.15.103:0/924562138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea0132030 0x7f2ea0134450 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.984+0000 7f2ea5b45700 1 -- 172.21.15.103:0/924562138 >> 172.21.15.103:0/924562138 conn(0x7f2ea009d810 msgr2=0x7f2ea009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.984+0000 7f2ea5b45700 1 -- 172.21.15.103:0/924562138 shutdown_connections 2023-07-02T09:03:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.984+0000 7f2ea5b45700 1 -- 172.21.15.103:0/924562138 wait complete. 2023-07-02T09:03:27.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.984+0000 7f2ea5b45700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.985+0000 7f2ea5b45700 1 Processor -- start 2023-07-02T09:03:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.985+0000 7f2ea5b45700 1 -- start start 2023-07-02T09:03:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2ea5b45700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea012f250 0x7f2ea006fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2ea5b45700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea0118610 con 0x7f2ea012f250 2023-07-02T09:03:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2e9f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea012f250 0x7f2ea006fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2e9f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea012f250 0x7f2ea006fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42672/0 (socket says 172.21.15.103:42672) 2023-07-02T09:03:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2e9f7fe700 1 -- 172.21.15.103:0/3520126949 learned_addr learned my addr 172.21.15.103:0/3520126949 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:27.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2e9f7fe700 1 -- 172.21.15.103:0/3520126949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea0117560 con 0x7f2ea012f250 2023-07-02T09:03:27.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2e9f7fe700 1 --2- 172.21.15.103:0/3520126949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea012f250 0x7f2ea006fe40 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:27.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2e97fff700 1 -- 172.21.15.103:0/3520126949 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2e88004b80 con 0x7f2ea012f250 2023-07-02T09:03:27.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2e97fff700 1 -- 172.21.15.103:0/3520126949 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2e88004ce0 con 0x7f2ea012f250 2023-07-02T09:03:27.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2e97fff700 1 -- 172.21.15.103:0/3520126949 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2e8800c4f0 con 0x7f2ea012f250 2023-07-02T09:03:27.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea0071aa0 con 0x7f2ea012f250 2023-07-02T09:03:27.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.986+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea0118fe0 con 0x7f2ea012f250 2023-07-02T09:03:27.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.987+0000 7f2e97fff700 1 -- 172.21.15.103:0/3520126949 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2e8800c650 con 0x7f2ea012f250 2023-07-02T09:03:27.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.987+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea0061ea0 con 0x7f2ea012f250 2023-07-02T09:03:27.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.987+0000 7f2e97fff700 1 --2- 172.21.15.103:0/3520126949 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2e800377c0 0x7f2e80039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:27.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.987+0000 7f2e97fff700 1 -- 172.21.15.103:0/3520126949 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2e88046990 con 0x7f2ea012f250 2023-07-02T09:03:27.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.988+0000 7f2e9effd700 1 --2- 172.21.15.103:0/3520126949 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2e800377c0 0x7f2e80039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:27.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.988+0000 7f2e9effd700 1 --2- 172.21.15.103:0/3520126949 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2e800377c0 0x7f2e80039cd0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:27.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:27.992+0000 7f2e97fff700 1 -- 172.21.15.103:0/3520126949 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2e88009c80 con 0x7f2ea012f250 2023-07-02T09:03:28.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.179+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2ea012c6c0 con 0x7f2ea012f250 2023-07-02T09:03:28.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.179+0000 7f2e97fff700 1 -- 172.21.15.103:0/3520126949 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2e880098a0 con 0x7f2ea012f250 2023-07-02T09:03:28.181 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:28.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2e800377c0 msgr2=0x7f2e80039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:28.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 --2- 172.21.15.103:0/3520126949 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2e800377c0 0x7f2e80039cd0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:28.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea012f250 msgr2=0x7f2ea006fe40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:28.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 --2- 172.21.15.103:0/3520126949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea012f250 0x7f2ea006fe40 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:28.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 shutdown_connections 2023-07-02T09:03:28.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 --2- 172.21.15.103:0/3520126949 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2e800377c0 0x7f2e80039cd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:28.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 --2- 172.21.15.103:0/3520126949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ea012f250 0x7f2ea006fe40 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:28.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 >> 172.21.15.103:0/3520126949 conn(0x7f2ea009d810 msgr2=0x7f2ea012e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:28.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 shutdown_connections 2023-07-02T09:03:28.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:28.181+0000 7f2ea5b45700 1 -- 172.21.15.103:0/3520126949 wait complete. 2023-07-02T09:03:28.195 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:29.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:29.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.329+0000 7f6520667700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:29.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.334+0000 7f6520667700 1 Processor -- start 2023-07-02T09:03:29.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.335+0000 7f6520667700 1 -- start start 2023-07-02T09:03:29.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.335+0000 7f6520667700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6518132f50 0x7f6518133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:29.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.335+0000 7f6520667700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651809e260 con 0x7f651809c580 2023-07-02T09:03:29.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.335+0000 7f6520667700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65180a1640 con 0x7f6518132f50 2023-07-02T09:03:29.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651e403700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6518132f50 0x7f6518133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:29.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651dc02700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f651809c580 0x7f651809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36552/0 (socket says 172.21.15.103:36552) 2023-07-02T09:03:29.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651dc02700 1 -- 172.21.15.103:0/2732962202 learned_addr learned my addr 172.21.15.103:0/2732962202 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:29.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651e403700 1 -- 172.21.15.103:0/2732962202 >> v1:172.21.15.103:6789/0 conn(0x7f651809c580 legacy=0x7f651809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:29.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651e403700 1 -- 172.21.15.103:0/2732962202 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6518132a20 con 0x7f6518132f50 2023-07-02T09:03:29.338 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651e403700 1 --2- 172.21.15.103:0/2732962202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6518132f50 0x7f6518133360 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beecc9995b9ea3e7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:29.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651d401700 1 -- 172.21.15.103:0/2732962202 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773506643 0 0) 0x7f651809e260 con 0x7f651809c580 2023-07-02T09:03:29.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651d401700 1 -- 172.21.15.103:0/2732962202 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6508009cd0 con 0x7f6518132f50 2023-07-02T09:03:29.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651d401700 1 -- 172.21.15.103:0/2732962202 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6508009e30 con 0x7f6518132f50 2023-07-02T09:03:29.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.336+0000 7f651d401700 1 -- 172.21.15.103:0/2732962202 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f650800e360 con 0x7f6518132f50 2023-07-02T09:03:29.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.337+0000 7f6520667700 1 -- 172.21.15.103:0/2732962202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6518132f50 msgr2=0x7f6518133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:29.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.337+0000 7f6520667700 1 --2- 172.21.15.103:0/2732962202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6518132f50 0x7f6518133360 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:29.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.337+0000 7f6520667700 1 -- 172.21.15.103:0/2732962202 shutdown_connections 2023-07-02T09:03:29.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.337+0000 7f6520667700 1 --2- 172.21.15.103:0/2732962202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6518132f50 0x7f6518133360 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:29.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.337+0000 7f6520667700 1 -- 172.21.15.103:0/2732962202 >> 172.21.15.103:0/2732962202 conn(0x7f65180a2080 msgr2=0x7f65180a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:29.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.337+0000 7f6520667700 1 -- 172.21.15.103:0/2732962202 shutdown_connections 2023-07-02T09:03:29.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.337+0000 7f6520667700 1 -- 172.21.15.103:0/2732962202 wait complete. 2023-07-02T09:03:29.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.338+0000 7f6520667700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:29.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.338+0000 7f6520667700 1 Processor -- start 2023-07-02T09:03:29.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.339+0000 7f6520667700 1 -- start start 2023-07-02T09:03:29.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.339+0000 7f6520667700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f651809c580 0x7f651812beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:29.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.339+0000 7f6520667700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6518133a00 con 0x7f651809c580 2023-07-02T09:03:29.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.339+0000 7f651e403700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f651809c580 0x7f651812beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:29.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.339+0000 7f651e403700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f651809c580 0x7f651812beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42700/0 (socket says 172.21.15.103:42700) 2023-07-02T09:03:29.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.339+0000 7f651e403700 1 -- 172.21.15.103:0/528032520 learned_addr learned my addr 172.21.15.103:0/528032520 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:29.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.340+0000 7f651e403700 1 -- 172.21.15.103:0/528032520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651806dc00 con 0x7f651809c580 2023-07-02T09:03:29.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.340+0000 7f651e403700 1 --2- 172.21.15.103:0/528032520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f651809c580 0x7f651812beb0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:29.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.340+0000 7f650effd700 1 -- 172.21.15.103:0/528032520 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6508014540 con 0x7f651809c580 2023-07-02T09:03:29.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.340+0000 7f650effd700 1 -- 172.21.15.103:0/528032520 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f65080146a0 con 0x7f651809c580 2023-07-02T09:03:29.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.340+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651806e920 con 0x7f651809c580 2023-07-02T09:03:29.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.340+0000 7f650effd700 1 -- 172.21.15.103:0/528032520 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6508014940 con 0x7f651809c580 2023-07-02T09:03:29.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.340+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6518071f80 con 0x7f651809c580 2023-07-02T09:03:29.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.341+0000 7f650effd700 1 -- 172.21.15.103:0/528032520 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6508014af0 con 0x7f651809c580 2023-07-02T09:03:29.344 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.341+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6518061ea0 con 0x7f651809c580 2023-07-02T09:03:29.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.341+0000 7f650effd700 1 --2- 172.21.15.103:0/528032520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f64fc0377c0 0x7f64fc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:29.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.341+0000 7f650effd700 1 -- 172.21.15.103:0/528032520 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6508048340 con 0x7f651809c580 2023-07-02T09:03:29.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.341+0000 7f651dc02700 1 --2- 172.21.15.103:0/528032520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f64fc0377c0 0x7f64fc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:29.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.342+0000 7f651dc02700 1 --2- 172.21.15.103:0/528032520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f64fc0377c0 0x7f64fc039cd0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:29.346 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.346+0000 7f650effd700 1 -- 172.21.15.103:0/528032520 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f650800fc20 con 0x7f651809c580 2023-07-02T09:03:29.532 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.531+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f65180725b0 con 0x7f651809c580 2023-07-02T09:03:29.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.531+0000 7f650effd700 1 -- 172.21.15.103:0/528032520 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6508011320 con 0x7f651809c580 2023-07-02T09:03:29.533 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:29.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f64fc0377c0 msgr2=0x7f64fc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:29.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 --2- 172.21.15.103:0/528032520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f64fc0377c0 0x7f64fc039cd0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:29.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f651809c580 msgr2=0x7f651812beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:29.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 --2- 172.21.15.103:0/528032520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f651809c580 0x7f651812beb0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 shutdown_connections 2023-07-02T09:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 --2- 172.21.15.103:0/528032520 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f64fc0377c0 0x7f64fc039cd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 --2- 172.21.15.103:0/528032520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f651809c580 0x7f651812beb0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 >> 172.21.15.103:0/528032520 conn(0x7f65180a2080 msgr2=0x7f65181ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 shutdown_connections 2023-07-02T09:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:29.534+0000 7f6520667700 1 -- 172.21.15.103:0/528032520 wait complete. 2023-07-02T09:03:29.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:30.549 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:30.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.686+0000 7f67e550b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:30.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.691+0000 7f67e550b700 1 Processor -- start 2023-07-02T09:03:30.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.691+0000 7f67e550b700 1 -- start start 2023-07-02T09:03:30.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67e550b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f67e012f250 0x7f67e012f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:30.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67e550b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e006df60 con 0x7f67e0098a00 2023-07-02T09:03:30.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67e550b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e009a710 con 0x7f67e012f250 2023-07-02T09:03:30.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67deffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f67e012f250 0x7f67e012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:30.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67de7fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f67e0098a00 0x7f67e0098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36568/0 (socket says 172.21.15.103:36568) 2023-07-02T09:03:30.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67de7fc700 1 -- 172.21.15.103:0/1510458582 learned_addr learned my addr 172.21.15.103:0/1510458582 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:30.695 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67deffd700 1 -- 172.21.15.103:0/1510458582 >> v1:172.21.15.103:6789/0 conn(0x7f67e0098a00 legacy=0x7f67e0098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:30.695 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67ddffb700 1 -- 172.21.15.103:0/1510458582 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527814368 0 0) 0x7f67e006df60 con 0x7f67e0098a00 2023-07-02T09:03:30.695 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.692+0000 7f67deffd700 1 -- 172.21.15.103:0/1510458582 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e012ed20 con 0x7f67e012f250 2023-07-02T09:03:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67deffd700 1 --2- 172.21.15.103:0/1510458582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e012f250 0x7f67e012f660 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f13fc60b61890d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67ddffb700 1 -- 172.21.15.103:0/1510458582 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67d0009cd0 con 0x7f67e012f250 2023-07-02T09:03:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67ddffb700 1 -- 172.21.15.103:0/1510458582 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f67d0009e30 con 0x7f67e012f250 2023-07-02T09:03:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67ddffb700 1 -- 172.21.15.103:0/1510458582 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67d000e360 con 0x7f67e012f250 2023-07-02T09:03:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67e550b700 1 -- 172.21.15.103:0/1510458582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e012f250 msgr2=0x7f67e012f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67e550b700 1 --2- 172.21.15.103:0/1510458582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e012f250 0x7f67e012f660 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67e550b700 1 -- 172.21.15.103:0/1510458582 shutdown_connections 2023-07-02T09:03:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67e550b700 1 --2- 172.21.15.103:0/1510458582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e012f250 0x7f67e012f660 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67e550b700 1 -- 172.21.15.103:0/1510458582 >> 172.21.15.103:0/1510458582 conn(0x7f67e009d810 msgr2=0x7f67e009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67e550b700 1 -- 172.21.15.103:0/1510458582 shutdown_connections 2023-07-02T09:03:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.693+0000 7f67e550b700 1 -- 172.21.15.103:0/1510458582 wait complete. 2023-07-02T09:03:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.694+0000 7f67e550b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.695+0000 7f67e550b700 1 Processor -- start 2023-07-02T09:03:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.695+0000 7f67e550b700 1 -- start start 2023-07-02T09:03:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.695+0000 7f67e550b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e0098a00 0x7f67e01181b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.695+0000 7f67e550b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e011f940 con 0x7f67e0098a00 2023-07-02T09:03:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67deffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e0098a00 0x7f67e01181b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67deffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e0098a00 0x7f67e01181b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42724/0 (socket says 172.21.15.103:42724) 2023-07-02T09:03:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67deffd700 1 -- 172.21.15.103:0/3680604695 learned_addr learned my addr 172.21.15.103:0/3680604695 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67deffd700 1 -- 172.21.15.103:0/3680604695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e0099af0 con 0x7f67e0098a00 2023-07-02T09:03:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67deffd700 1 --2- 172.21.15.103:0/3680604695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e0098a00 0x7f67e01181b0 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67d77fe700 1 -- 172.21.15.103:0/3680604695 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67d0014540 con 0x7f67e0098a00 2023-07-02T09:03:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67d77fe700 1 -- 172.21.15.103:0/3680604695 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f67d00146a0 con 0x7f67e0098a00 2023-07-02T09:03:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e011f630 con 0x7f67e0098a00 2023-07-02T09:03:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67d77fe700 1 -- 172.21.15.103:0/3680604695 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67d0014940 con 0x7f67e0098a00 2023-07-02T09:03:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.696+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e0119680 con 0x7f67e0098a00 2023-07-02T09:03:30.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.697+0000 7f67d77fe700 1 -- 172.21.15.103:0/3680604695 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f67d0014af0 con 0x7f67e0098a00 2023-07-02T09:03:30.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.697+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e0061ea0 con 0x7f67e0098a00 2023-07-02T09:03:30.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.697+0000 7f67d77fe700 1 --2- 172.21.15.103:0/3680604695 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67c00377b0 0x7f67c0039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:30.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.697+0000 7f67d77fe700 1 -- 172.21.15.103:0/3680604695 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f67d0048270 con 0x7f67e0098a00 2023-07-02T09:03:30.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.698+0000 7f67de7fc700 1 --2- 172.21.15.103:0/3680604695 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67c00377b0 0x7f67c0039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:30.701 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.698+0000 7f67de7fc700 1 --2- 172.21.15.103:0/3680604695 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67c00377b0 0x7f67c0039cc0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.702+0000 7f67d77fe700 1 -- 172.21.15.103:0/3680604695 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f67d000fe40 con 0x7f67e0098a00 2023-07-02T09:03:30.889 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.888+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f67e0073ca0 con 0x7f67e0098a00 2023-07-02T09:03:30.889 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.888+0000 7f67d77fe700 1 -- 172.21.15.103:0/3680604695 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f67d000fa50 con 0x7f67e0098a00 2023-07-02T09:03:30.890 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:30.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67c00377b0 msgr2=0x7f67c0039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:30.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 --2- 172.21.15.103:0/3680604695 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67c00377b0 0x7f67c0039cc0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:30.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e0098a00 msgr2=0x7f67e01181b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:30.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 --2- 172.21.15.103:0/3680604695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e0098a00 0x7f67e01181b0 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:30.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 shutdown_connections 2023-07-02T09:03:30.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 --2- 172.21.15.103:0/3680604695 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67c00377b0 0x7f67c0039cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:30.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 --2- 172.21.15.103:0/3680604695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67e0098a00 0x7f67e01181b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:30.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 >> 172.21.15.103:0/3680604695 conn(0x7f67e009d810 msgr2=0x7f67e009eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:30.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 shutdown_connections 2023-07-02T09:03:30.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:30.890+0000 7f67e550b700 1 -- 172.21.15.103:0/3680604695 wait complete. 2023-07-02T09:03:30.904 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:31.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~226s 2023-07-02T09:03:31.904 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:32.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.042+0000 7fdf40df7700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:32.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.046+0000 7fdf40df7700 1 Processor -- start 2023-07-02T09:03:32.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.047+0000 7fdf40df7700 1 -- start start 2023-07-02T09:03:32.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.047+0000 7fdf40df7700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf3c133360 0x7fdf3c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:32.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.047+0000 7fdf40df7700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c09e260 con 0x7fdf3c132f50 2023-07-02T09:03:32.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.047+0000 7fdf40df7700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3c0a1640 con 0x7fdf3c133360 2023-07-02T09:03:32.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf39d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf3c133360 0x7fdf3c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:32.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf3a59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdf3c132f50 0x7fdf3c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:36582/0 (socket says 172.21.15.103:36582) 2023-07-02T09:03:32.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf3a59c700 1 -- 172.21.15.103:0/3123940265 learned_addr learned my addr 172.21.15.103:0/3123940265 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:32.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf39d9b700 1 -- 172.21.15.103:0/3123940265 >> v1:172.21.15.103:6789/0 conn(0x7fdf3c132f50 legacy=0x7fdf3c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:32.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf3959a700 1 -- 172.21.15.103:0/3123940265 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698710635 0 0) 0x7fdf3c09e260 con 0x7fdf3c132f50 2023-07-02T09:03:32.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf39d9b700 1 -- 172.21.15.103:0/3123940265 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c132a20 con 0x7fdf3c133360 2023-07-02T09:03:32.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf39d9b700 1 --2- 172.21.15.103:0/3123940265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c133360 0x7fdf3c09cc90 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878b3f236f91c157 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:32.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf3959a700 1 -- 172.21.15.103:0/3123940265 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf2c009cd0 con 0x7fdf3c133360 2023-07-02T09:03:32.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf3959a700 1 -- 172.21.15.103:0/3123940265 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdf2c009e30 con 0x7fdf3c133360 2023-07-02T09:03:32.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.048+0000 7fdf3959a700 1 -- 172.21.15.103:0/3123940265 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf2c00e360 con 0x7fdf3c133360 2023-07-02T09:03:32.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.049+0000 7fdf40df7700 1 -- 172.21.15.103:0/3123940265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c133360 msgr2=0x7fdf3c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:32.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.049+0000 7fdf40df7700 1 --2- 172.21.15.103:0/3123940265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c133360 0x7fdf3c09cc90 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:32.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.049+0000 7fdf40df7700 1 -- 172.21.15.103:0/3123940265 shutdown_connections 2023-07-02T09:03:32.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.049+0000 7fdf40df7700 1 --2- 172.21.15.103:0/3123940265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c133360 0x7fdf3c09cc90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:32.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.049+0000 7fdf40df7700 1 -- 172.21.15.103:0/3123940265 >> 172.21.15.103:0/3123940265 conn(0x7fdf3c0a2080 msgr2=0x7fdf3c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:32.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.049+0000 7fdf40df7700 1 -- 172.21.15.103:0/3123940265 shutdown_connections 2023-07-02T09:03:32.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.049+0000 7fdf40df7700 1 -- 172.21.15.103:0/3123940265 wait complete. 2023-07-02T09:03:32.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.049+0000 7fdf40df7700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:32.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.050+0000 7fdf40df7700 1 Processor -- start 2023-07-02T09:03:32.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.050+0000 7fdf40df7700 1 -- start start 2023-07-02T09:03:32.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.051+0000 7fdf40df7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c132f50 0x7fdf3c1ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:32.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.051+0000 7fdf40df7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c06d370 con 0x7fdf3c132f50 2023-07-02T09:03:32.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.051+0000 7fdf3a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c132f50 0x7fdf3c1ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:32.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.051+0000 7fdf3a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c132f50 0x7fdf3c1ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42742/0 (socket says 172.21.15.103:42742) 2023-07-02T09:03:32.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.051+0000 7fdf3a59c700 1 -- 172.21.15.103:0/2548089252 learned_addr learned my addr 172.21.15.103:0/2548089252 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:32.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.051+0000 7fdf3a59c700 1 -- 172.21.15.103:0/2548089252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c06dde0 con 0x7fdf3c132f50 2023-07-02T09:03:32.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.052+0000 7fdf3a59c700 1 --2- 172.21.15.103:0/2548089252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c132f50 0x7fdf3c1ba9b0 crc :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:32.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.052+0000 7fdf2affd700 1 -- 172.21.15.103:0/2548089252 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf30004b80 con 0x7fdf3c132f50 2023-07-02T09:03:32.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.052+0000 7fdf2affd700 1 -- 172.21.15.103:0/2548089252 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdf30004ce0 con 0x7fdf3c132f50 2023-07-02T09:03:32.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.052+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c1ba4f0 con 0x7fdf3c132f50 2023-07-02T09:03:32.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.052+0000 7fdf2affd700 1 -- 172.21.15.103:0/2548089252 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdf3000c4f0 con 0x7fdf3c132f50 2023-07-02T09:03:32.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.052+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c071e40 con 0x7fdf3c132f50 2023-07-02T09:03:32.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.052+0000 7fdf2affd700 1 -- 172.21.15.103:0/2548089252 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fdf3000c650 con 0x7fdf3c132f50 2023-07-02T09:03:32.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.053+0000 7fdf2affd700 1 --2- 172.21.15.103:0/2548089252 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf1c0377c0 0x7fdf1c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:32.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.053+0000 7fdf2affd700 1 -- 172.21.15.103:0/2548089252 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fdf30046970 con 0x7fdf3c132f50 2023-07-02T09:03:32.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.053+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3c061ea0 con 0x7fdf3c132f50 2023-07-02T09:03:32.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.053+0000 7fdf39d9b700 1 --2- 172.21.15.103:0/2548089252 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf1c0377c0 0x7fdf1c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:32.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.054+0000 7fdf39d9b700 1 --2- 172.21.15.103:0/2548089252 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf1c0377c0 0x7fdf1c039cd0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:32.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.059+0000 7fdf2affd700 1 -- 172.21.15.103:0/2548089252 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fdf30009c80 con 0x7fdf3c132f50 2023-07-02T09:03:32.246 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.245+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdf3c0725b0 con 0x7fdf3c132f50 2023-07-02T09:03:32.246 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.245+0000 7fdf2affd700 1 -- 172.21.15.103:0/2548089252 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fdf3002c020 con 0x7fdf3c132f50 2023-07-02T09:03:32.246 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:32.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf1c0377c0 msgr2=0x7fdf1c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:32.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 --2- 172.21.15.103:0/2548089252 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf1c0377c0 0x7fdf1c039cd0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:32.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c132f50 msgr2=0x7fdf3c1ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:32.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 --2- 172.21.15.103:0/2548089252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c132f50 0x7fdf3c1ba9b0 crc :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:32.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 shutdown_connections 2023-07-02T09:03:32.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 --2- 172.21.15.103:0/2548089252 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fdf1c0377c0 0x7fdf1c039cd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:32.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 --2- 172.21.15.103:0/2548089252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf3c132f50 0x7fdf3c1ba9b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:32.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 >> 172.21.15.103:0/2548089252 conn(0x7fdf3c0a2080 msgr2=0x7fdf3c0a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:32.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 shutdown_connections 2023-07-02T09:03:32.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:32.247+0000 7fdf40df7700 1 -- 172.21.15.103:0/2548089252 wait complete. 2023-07-02T09:03:32.261 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:33.262 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:33.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.398+0000 7fcc06d90700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:33.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.403+0000 7fcc06d90700 1 Processor -- start 2023-07-02T09:03:33.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.403+0000 7fcc06d90700 1 -- start start 2023-07-02T09:03:33.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcc06d90700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcc00132f50 0x7fcc00133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:33.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcc06d90700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0009e260 con 0x7fcc0009c580 2023-07-02T09:03:33.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcc06d90700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc000a1640 con 0x7fcc00132f50 2023-07-02T09:03:33.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcc04b2c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcc00132f50 0x7fcc00133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:33.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcbfffff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fcc0009c580 0x7fcc0009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37228/0 (socket says 172.21.15.103:37228) 2023-07-02T09:03:33.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcbfffff700 1 -- 172.21.15.103:0/2596420478 learned_addr learned my addr 172.21.15.103:0/2596420478 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:33.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcc04b2c700 1 -- 172.21.15.103:0/2596420478 >> v1:172.21.15.103:6789/0 conn(0x7fcc0009c580 legacy=0x7fcc0009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:33.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcc04b2c700 1 -- 172.21.15.103:0/2596420478 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc00132a20 con 0x7fcc00132f50 2023-07-02T09:03:33.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcbff7fe700 1 -- 172.21.15.103:0/2596420478 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882732411 0 0) 0x7fcc0009e260 con 0x7fcc0009c580 2023-07-02T09:03:33.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcc04b2c700 1 --2- 172.21.15.103:0/2596420478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc00132f50 0x7fcc00133360 crc :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f72647a0085cb1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:33.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcbff7fe700 1 -- 172.21.15.103:0/2596420478 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fcbf4009cd0 con 0x7fcc00132f50 2023-07-02T09:03:33.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcbff7fe700 1 -- 172.21.15.103:0/2596420478 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fcbf4009e30 con 0x7fcc00132f50 2023-07-02T09:03:33.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.404+0000 7fcbff7fe700 1 -- 172.21.15.103:0/2596420478 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fcbf400e360 con 0x7fcc00132f50 2023-07-02T09:03:33.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.405+0000 7fcc06d90700 1 -- 172.21.15.103:0/2596420478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc00132f50 msgr2=0x7fcc00133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:33.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.405+0000 7fcc06d90700 1 --2- 172.21.15.103:0/2596420478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc00132f50 0x7fcc00133360 crc :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:33.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.405+0000 7fcc06d90700 1 -- 172.21.15.103:0/2596420478 shutdown_connections 2023-07-02T09:03:33.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.405+0000 7fcc06d90700 1 --2- 172.21.15.103:0/2596420478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc00132f50 0x7fcc00133360 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:33.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.405+0000 7fcc06d90700 1 -- 172.21.15.103:0/2596420478 >> 172.21.15.103:0/2596420478 conn(0x7fcc000a2080 msgr2=0x7fcc000a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:33.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.405+0000 7fcc06d90700 1 -- 172.21.15.103:0/2596420478 shutdown_connections 2023-07-02T09:03:33.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.405+0000 7fcc06d90700 1 -- 172.21.15.103:0/2596420478 wait complete. 2023-07-02T09:03:33.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.405+0000 7fcc06d90700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:33.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.406+0000 7fcc06d90700 1 Processor -- start 2023-07-02T09:03:33.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.406+0000 7fcc06d90700 1 -- start start 2023-07-02T09:03:33.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.407+0000 7fcc06d90700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0009c580 0x7fcc001baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:33.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.407+0000 7fcc06d90700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc001ba610 con 0x7fcc0009c580 2023-07-02T09:03:33.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.407+0000 7fcc04b2c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0009c580 0x7fcc001baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:33.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.407+0000 7fcc04b2c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0009c580 0x7fcc001baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58204/0 (socket says 172.21.15.103:58204) 2023-07-02T09:03:33.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.407+0000 7fcc04b2c700 1 -- 172.21.15.103:0/594678943 learned_addr learned my addr 172.21.15.103:0/594678943 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:33.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.407+0000 7fcc04b2c700 1 -- 172.21.15.103:0/594678943 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0006dfc0 con 0x7fcc0009c580 2023-07-02T09:03:33.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.408+0000 7fcc04b2c700 1 --2- 172.21.15.103:0/594678943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0009c580 0x7fcc001baa60 crc :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:33.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.408+0000 7fcbfd7fa700 1 -- 172.21.15.103:0/594678943 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fcbf4013530 con 0x7fcc0009c580 2023-07-02T09:03:33.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.408+0000 7fcbfd7fa700 1 -- 172.21.15.103:0/594678943 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fcbf4013690 con 0x7fcc0009c580 2023-07-02T09:03:33.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.408+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc001ba270 con 0x7fcc0009c580 2023-07-02T09:03:33.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.408+0000 7fcbfd7fa700 1 -- 172.21.15.103:0/594678943 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fcbf4013930 con 0x7fcc0009c580 2023-07-02T09:03:33.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.408+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0006dba0 con 0x7fcc0009c580 2023-07-02T09:03:33.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.409+0000 7fcbfd7fa700 1 -- 172.21.15.103:0/594678943 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fcbf4013b30 con 0x7fcc0009c580 2023-07-02T09:03:33.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.409+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc001ba460 con 0x7fcc0009c580 2023-07-02T09:03:33.412 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.409+0000 7fcbfd7fa700 1 --2- 172.21.15.103:0/594678943 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fcbe4037760 0x7fcbe4039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:33.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.409+0000 7fcbfd7fa700 1 -- 172.21.15.103:0/594678943 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fcbf4018040 con 0x7fcc0009c580 2023-07-02T09:03:33.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.409+0000 7fcbfffff700 1 --2- 172.21.15.103:0/594678943 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fcbe4037760 0x7fcbe4039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:33.413 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.410+0000 7fcbfffff700 1 --2- 172.21.15.103:0/594678943 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fcbe4037760 0x7fcbe4039c70 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:33.414 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.414+0000 7fcbfd7fa700 1 -- 172.21.15.103:0/594678943 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fcbf4018e00 con 0x7fcc0009c580 2023-07-02T09:03:33.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.600+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fcc00061ea0 con 0x7fcc0009c580 2023-07-02T09:03:33.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.601+0000 7fcbfd7fa700 1 -- 172.21.15.103:0/594678943 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fcbf4010e60 con 0x7fcc0009c580 2023-07-02T09:03:33.601 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:33.603 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fcbe4037760 msgr2=0x7fcbe4039c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:33.603 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 --2- 172.21.15.103:0/594678943 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fcbe4037760 0x7fcbe4039c70 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:33.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0009c580 msgr2=0x7fcc001baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:33.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 --2- 172.21.15.103:0/594678943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0009c580 0x7fcc001baa60 crc :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:33.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 shutdown_connections 2023-07-02T09:03:33.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 --2- 172.21.15.103:0/594678943 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fcbe4037760 0x7fcbe4039c70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:33.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 --2- 172.21.15.103:0/594678943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0009c580 0x7fcc001baa60 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:33.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 >> 172.21.15.103:0/594678943 conn(0x7fcc000a2080 msgr2=0x7fcc00130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:33.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 shutdown_connections 2023-07-02T09:03:33.605 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:33.603+0000 7fcc06d90700 1 -- 172.21.15.103:0/594678943 wait complete. 2023-07-02T09:03:33.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:34.617 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.756+0000 7fedd383d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.761+0000 7fedd383d700 1 Processor -- start 2023-07-02T09:03:34.762 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.761+0000 7fedd383d700 1 -- start start 2023-07-02T09:03:34.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedd383d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fedcc132030 0x7fedcc134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:34.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedd383d700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedcc06df60 con 0x7fedcc12f250 2023-07-02T09:03:34.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedd383d700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedcc09a710 con 0x7fedcc132030 2023-07-02T09:03:34.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedd15d9700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fedcc12f250 0x7fedcc12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37240/0 (socket says 172.21.15.103:37240) 2023-07-02T09:03:34.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedd15d9700 1 -- 172.21.15.103:0/31081835 learned_addr learned my addr 172.21.15.103:0/31081835 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:34.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedd0dd8700 1 --2- 172.21.15.103:0/31081835 >> v2:172.21.15.103:3300/0 conn(0x7fedcc132030 0x7fedcc134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:34.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedc3fff700 1 -- 172.21.15.103:0/31081835 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634422352 0 0) 0x7fedcc06df60 con 0x7fedcc12f250 2023-07-02T09:03:34.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedc3fff700 1 -- 172.21.15.103:0/31081835 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb80034a0 con 0x7fedcc12f250 2023-07-02T09:03:34.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.762+0000 7fedd0dd8700 1 -- 172.21.15.103:0/31081835 >> v1:172.21.15.103:6789/0 conn(0x7fedcc12f250 legacy=0x7fedcc12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:34.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedd0dd8700 1 -- 172.21.15.103:0/31081835 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc12ed20 con 0x7fedcc132030 2023-07-02T09:03:34.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedd0dd8700 1 --2- 172.21.15.103:0/31081835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc132030 0x7fedcc134450 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed55e4258ec17a9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:34.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedc3fff700 1 -- 172.21.15.103:0/31081835 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fedc8009c80 con 0x7fedcc132030 2023-07-02T09:03:34.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedc3fff700 1 -- 172.21.15.103:0/31081835 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fedc8009de0 con 0x7fedcc132030 2023-07-02T09:03:34.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedc3fff700 1 -- 172.21.15.103:0/31081835 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fedc800e360 con 0x7fedcc132030 2023-07-02T09:03:34.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedd383d700 1 -- 172.21.15.103:0/31081835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc132030 msgr2=0x7fedcc134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:34.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedd383d700 1 --2- 172.21.15.103:0/31081835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc132030 0x7fedcc134450 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:34.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedd383d700 1 -- 172.21.15.103:0/31081835 shutdown_connections 2023-07-02T09:03:34.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedd383d700 1 --2- 172.21.15.103:0/31081835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc132030 0x7fedcc134450 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:34.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.763+0000 7fedd383d700 1 -- 172.21.15.103:0/31081835 >> 172.21.15.103:0/31081835 conn(0x7fedcc09d810 msgr2=0x7fedcc09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:34.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.764+0000 7fedd383d700 1 -- 172.21.15.103:0/31081835 shutdown_connections 2023-07-02T09:03:34.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.764+0000 7fedd383d700 1 -- 172.21.15.103:0/31081835 wait complete. 2023-07-02T09:03:34.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.764+0000 7fedd383d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:34.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.765+0000 7fedd383d700 1 Processor -- start 2023-07-02T09:03:34.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.765+0000 7fedd383d700 1 -- start start 2023-07-02T09:03:34.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.765+0000 7fedd383d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc12f250 0x7fedcc1c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:34.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.765+0000 7fedd383d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc071120 con 0x7fedcc12f250 2023-07-02T09:03:34.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.765+0000 7fedd15d9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc12f250 0x7fedcc1c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:34.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.765+0000 7fedd15d9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc12f250 0x7fedcc1c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58218/0 (socket says 172.21.15.103:58218) 2023-07-02T09:03:34.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.765+0000 7fedd15d9700 1 -- 172.21.15.103:0/3375298264 learned_addr learned my addr 172.21.15.103:0/3375298264 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:34.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.766+0000 7fedd15d9700 1 -- 172.21.15.103:0/3375298264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc11b990 con 0x7fedcc12f250 2023-07-02T09:03:34.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.766+0000 7fedd15d9700 1 --2- 172.21.15.103:0/3375298264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc12f250 0x7fedcc1c7250 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:34.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.766+0000 7fedc1ffb700 1 -- 172.21.15.103:0/3375298264 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fedbc005a60 con 0x7fedcc12f250 2023-07-02T09:03:34.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.766+0000 7fedc1ffb700 1 -- 172.21.15.103:0/3375298264 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fedbc005bc0 con 0x7fedcc12f250 2023-07-02T09:03:34.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.766+0000 7fedc1ffb700 1 -- 172.21.15.103:0/3375298264 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fedbc005e60 con 0x7fedcc12f250 2023-07-02T09:03:34.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.766+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedcc11c9b0 con 0x7fedcc12f250 2023-07-02T09:03:34.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.766+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedcc128d30 con 0x7fedcc12f250 2023-07-02T09:03:34.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.767+0000 7fedc1ffb700 1 -- 172.21.15.103:0/3375298264 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fedbc0070b0 con 0x7fedcc12f250 2023-07-02T09:03:34.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.767+0000 7fedc1ffb700 1 --2- 172.21.15.103:0/3375298264 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fedb00377c0 0x7fedb0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:34.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.767+0000 7fedc1ffb700 1 -- 172.21.15.103:0/3375298264 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fedbc046ba0 con 0x7fedcc12f250 2023-07-02T09:03:34.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.767+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedcc061ea0 con 0x7fedcc12f250 2023-07-02T09:03:34.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.767+0000 7fedd0dd8700 1 --2- 172.21.15.103:0/3375298264 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fedb00377c0 0x7fedb0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:34.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.768+0000 7fedd0dd8700 1 --2- 172.21.15.103:0/3375298264 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fedb00377c0 0x7fedb0039cd0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:34.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.772+0000 7fedc1ffb700 1 -- 172.21.15.103:0/3375298264 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fedbc00a6f0 con 0x7fedcc12f250 2023-07-02T09:03:34.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.957+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fedcc12c6c0 con 0x7fedcc12f250 2023-07-02T09:03:34.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.958+0000 7fedc1ffb700 1 -- 172.21.15.103:0/3375298264 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fedbc00a120 con 0x7fedcc12f250 2023-07-02T09:03:34.959 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:34.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fedb00377c0 msgr2=0x7fedb0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:34.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 --2- 172.21.15.103:0/3375298264 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fedb00377c0 0x7fedb0039cd0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:34.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc12f250 msgr2=0x7fedcc1c7250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:34.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 --2- 172.21.15.103:0/3375298264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc12f250 0x7fedcc1c7250 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:34.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 shutdown_connections 2023-07-02T09:03:34.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 --2- 172.21.15.103:0/3375298264 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fedb00377c0 0x7fedb0039cd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:34.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 --2- 172.21.15.103:0/3375298264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedcc12f250 0x7fedcc1c7250 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:34.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 >> 172.21.15.103:0/3375298264 conn(0x7fedcc09d810 msgr2=0x7fedcc12dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:34.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 shutdown_connections 2023-07-02T09:03:34.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:34.960+0000 7fedd383d700 1 -- 172.21.15.103:0/3375298264 wait complete. 2023-07-02T09:03:34.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:35.974 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:36.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.112+0000 7fe920f5b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:36.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.117+0000 7fe920f5b700 1 Processor -- start 2023-07-02T09:03:36.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.118+0000 7fe920f5b700 1 -- start start 2023-07-02T09:03:36.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.118+0000 7fe920f5b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe91c132030 0x7fe91c134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:36.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.118+0000 7fe920f5b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c06df60 con 0x7fe91c12f250 2023-07-02T09:03:36.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.118+0000 7fe920f5b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91c09a710 con 0x7fe91c132030 2023-07-02T09:03:36.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe919d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe91c132030 0x7fe91c134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:36.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe91a59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe91c12f250 0x7fe91c12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37254/0 (socket says 172.21.15.103:37254) 2023-07-02T09:03:36.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe91a59c700 1 -- 172.21.15.103:0/363557573 learned_addr learned my addr 172.21.15.103:0/363557573 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:36.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe919d9b700 1 -- 172.21.15.103:0/363557573 >> v1:172.21.15.103:6789/0 conn(0x7fe91c12f250 legacy=0x7fe91c12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:36.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe919d9b700 1 -- 172.21.15.103:0/363557573 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c12ed20 con 0x7fe91c132030 2023-07-02T09:03:36.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe91959a700 1 -- 172.21.15.103:0/363557573 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426978848 0 0) 0x7fe91c06df60 con 0x7fe91c12f250 2023-07-02T09:03:36.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe919d9b700 1 --2- 172.21.15.103:0/363557573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c132030 0x7fe91c134450 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9578794ddb241f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:36.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe91959a700 1 -- 172.21.15.103:0/363557573 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe90c00ac60 con 0x7fe91c132030 2023-07-02T09:03:36.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe91959a700 1 -- 172.21.15.103:0/363557573 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe90c00adc0 con 0x7fe91c132030 2023-07-02T09:03:36.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.119+0000 7fe91959a700 1 -- 172.21.15.103:0/363557573 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe90c00f470 con 0x7fe91c132030 2023-07-02T09:03:36.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.120+0000 7fe920f5b700 1 -- 172.21.15.103:0/363557573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c132030 msgr2=0x7fe91c134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:36.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.120+0000 7fe920f5b700 1 --2- 172.21.15.103:0/363557573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c132030 0x7fe91c134450 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:36.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.120+0000 7fe920f5b700 1 -- 172.21.15.103:0/363557573 shutdown_connections 2023-07-02T09:03:36.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.120+0000 7fe920f5b700 1 --2- 172.21.15.103:0/363557573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c132030 0x7fe91c134450 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:36.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.120+0000 7fe920f5b700 1 -- 172.21.15.103:0/363557573 >> 172.21.15.103:0/363557573 conn(0x7fe91c09d810 msgr2=0x7fe91c09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:36.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.120+0000 7fe920f5b700 1 -- 172.21.15.103:0/363557573 shutdown_connections 2023-07-02T09:03:36.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.120+0000 7fe920f5b700 1 -- 172.21.15.103:0/363557573 wait complete. 2023-07-02T09:03:36.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.120+0000 7fe920f5b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:36.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.121+0000 7fe920f5b700 1 Processor -- start 2023-07-02T09:03:36.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.121+0000 7fe920f5b700 1 -- start start 2023-07-02T09:03:36.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.122+0000 7fe920f5b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c12f250 0x7fe91c06fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:36.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.122+0000 7fe920f5b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c071390 con 0x7fe91c12f250 2023-07-02T09:03:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.122+0000 7fe91a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c12f250 0x7fe91c06fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.122+0000 7fe91a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c12f250 0x7fe91c06fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58240/0 (socket says 172.21.15.103:58240) 2023-07-02T09:03:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.122+0000 7fe91a59c700 1 -- 172.21.15.103:0/3327477337 learned_addr learned my addr 172.21.15.103:0/3327477337 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.122+0000 7fe91a59c700 1 -- 172.21.15.103:0/3327477337 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c1292e0 con 0x7fe91c12f250 2023-07-02T09:03:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.122+0000 7fe91a59c700 1 --2- 172.21.15.103:0/3327477337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c12f250 0x7fe91c06fe70 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.123+0000 7fe90affd700 1 -- 172.21.15.103:0/3327477337 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe910004b80 con 0x7fe91c12f250 2023-07-02T09:03:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.123+0000 7fe90affd700 1 -- 172.21.15.103:0/3327477337 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe910004ce0 con 0x7fe91c12f250 2023-07-02T09:03:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.123+0000 7fe90affd700 1 -- 172.21.15.103:0/3327477337 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe91000c4f0 con 0x7fe91c12f250 2023-07-02T09:03:36.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.123+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c128f40 con 0x7fe91c12f250 2023-07-02T09:03:36.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.123+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c1179c0 con 0x7fe91c12f250 2023-07-02T09:03:36.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.124+0000 7fe90affd700 1 -- 172.21.15.103:0/3327477337 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fe91000c650 con 0x7fe91c12f250 2023-07-02T09:03:36.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.124+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91c061ea0 con 0x7fe91c12f250 2023-07-02T09:03:36.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.124+0000 7fe90affd700 1 --2- 172.21.15.103:0/3327477337 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8fc0377c0 0x7fe8fc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:36.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.124+0000 7fe90affd700 1 -- 172.21.15.103:0/3327477337 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fe9100467f0 con 0x7fe91c12f250 2023-07-02T09:03:36.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.124+0000 7fe919d9b700 1 --2- 172.21.15.103:0/3327477337 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8fc0377c0 0x7fe8fc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:36.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.124+0000 7fe919d9b700 1 --2- 172.21.15.103:0/3327477337 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8fc0377c0 0x7fe8fc039cd0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:36.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.129+0000 7fe90affd700 1 -- 172.21.15.103:0/3327477337 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fe910009ae0 con 0x7fe91c12f250 2023-07-02T09:03:36.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.314+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe91c1292c0 con 0x7fe91c12f250 2023-07-02T09:03:36.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.315+0000 7fe90affd700 1 -- 172.21.15.103:0/3327477337 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fe9100090b0 con 0x7fe91c12f250 2023-07-02T09:03:36.315 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:36.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8fc0377c0 msgr2=0x7fe8fc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:36.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 --2- 172.21.15.103:0/3327477337 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8fc0377c0 0x7fe8fc039cd0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:36.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c12f250 msgr2=0x7fe91c06fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:36.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 --2- 172.21.15.103:0/3327477337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c12f250 0x7fe91c06fe70 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:36.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 shutdown_connections 2023-07-02T09:03:36.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 --2- 172.21.15.103:0/3327477337 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe8fc0377c0 0x7fe8fc039cd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:36.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 --2- 172.21.15.103:0/3327477337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe91c12f250 0x7fe91c06fe70 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:36.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 >> 172.21.15.103:0/3327477337 conn(0x7fe91c09d810 msgr2=0x7fe91c12dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:36.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 shutdown_connections 2023-07-02T09:03:36.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:36.317+0000 7fe920f5b700 1 -- 172.21.15.103:0/3327477337 wait complete. 2023-07-02T09:03:36.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:37.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~231s 2023-07-02T09:03:37.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:37.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.470+0000 7f116e854700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:37.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.475+0000 7f116e854700 1 Processor -- start 2023-07-02T09:03:37.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.475+0000 7f116e854700 1 -- start start 2023-07-02T09:03:37.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f116e854700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1168132030 0x7f1168134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:37.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f116e854700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116806df60 con 0x7f116812f250 2023-07-02T09:03:37.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f116e854700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f116809a710 con 0x7f1168132030 2023-07-02T09:03:37.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f11677fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1168132030 0x7f1168134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:37.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f1167fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f116812f250 0x7f116812f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37270/0 (socket says 172.21.15.103:37270) 2023-07-02T09:03:37.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f1167fff700 1 -- 172.21.15.103:0/4110568362 learned_addr learned my addr 172.21.15.103:0/4110568362 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:37.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f11677fe700 1 -- 172.21.15.103:0/4110568362 >> v1:172.21.15.103:6789/0 conn(0x7f116812f250 legacy=0x7f116812f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:37.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f11677fe700 1 -- 172.21.15.103:0/4110568362 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116812ed20 con 0x7f1168132030 2023-07-02T09:03:37.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f11677fe700 1 --2- 172.21.15.103:0/4110568362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1168132030 0x7f1168134450 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69f9502136474ce server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:37.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f1166ffd700 1 -- 172.21.15.103:0/4110568362 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1158009cd0 con 0x7f1168132030 2023-07-02T09:03:37.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f1166ffd700 1 -- 172.21.15.103:0/4110568362 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1158009e30 con 0x7f1168132030 2023-07-02T09:03:37.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.476+0000 7f1166ffd700 1 -- 172.21.15.103:0/4110568362 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f115800e360 con 0x7f1168132030 2023-07-02T09:03:37.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.477+0000 7f116e854700 1 -- 172.21.15.103:0/4110568362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1168132030 msgr2=0x7f1168134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:37.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.477+0000 7f116e854700 1 --2- 172.21.15.103:0/4110568362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1168132030 0x7f1168134450 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:37.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.477+0000 7f116e854700 1 -- 172.21.15.103:0/4110568362 shutdown_connections 2023-07-02T09:03:37.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.477+0000 7f116e854700 1 --2- 172.21.15.103:0/4110568362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1168132030 0x7f1168134450 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:37.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.477+0000 7f116e854700 1 -- 172.21.15.103:0/4110568362 >> 172.21.15.103:0/4110568362 conn(0x7f116809d810 msgr2=0x7f116809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:37.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.477+0000 7f116e854700 1 -- 172.21.15.103:0/4110568362 shutdown_connections 2023-07-02T09:03:37.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.477+0000 7f116e854700 1 -- 172.21.15.103:0/4110568362 wait complete. 2023-07-02T09:03:37.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.477+0000 7f116e854700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:37.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.478+0000 7f116e854700 1 Processor -- start 2023-07-02T09:03:37.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.478+0000 7f116e854700 1 -- start start 2023-07-02T09:03:37.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.479+0000 7f116e854700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f116812f250 0x7f11681c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:37.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.479+0000 7f116e854700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1168117360 con 0x7f116812f250 2023-07-02T09:03:37.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.479+0000 7f1167fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f116812f250 0x7f11681c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:37.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.479+0000 7f1167fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f116812f250 0x7f11681c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58248/0 (socket says 172.21.15.103:58248) 2023-07-02T09:03:37.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.479+0000 7f1167fff700 1 -- 172.21.15.103:0/3186940527 learned_addr learned my addr 172.21.15.103:0/3186940527 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:37.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.480+0000 7f1167fff700 1 -- 172.21.15.103:0/3186940527 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11681177b0 con 0x7f116812f250 2023-07-02T09:03:37.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.480+0000 7f1167fff700 1 --2- 172.21.15.103:0/3186940527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f116812f250 0x7f11681c3ac0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:37.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.480+0000 7f1164ff9700 1 -- 172.21.15.103:0/3186940527 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f115c005a60 con 0x7f116812f250 2023-07-02T09:03:37.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.480+0000 7f1164ff9700 1 -- 172.21.15.103:0/3186940527 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f115c005bc0 con 0x7f116812f250 2023-07-02T09:03:37.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.480+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116806fd40 con 0x7f116812f250 2023-07-02T09:03:37.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.480+0000 7f1164ff9700 1 -- 172.21.15.103:0/3186940527 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f115c005e60 con 0x7f116812f250 2023-07-02T09:03:37.483 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.480+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1168071900 con 0x7f116812f250 2023-07-02T09:03:37.484 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.481+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1168061ea0 con 0x7f116812f250 2023-07-02T09:03:37.484 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.481+0000 7f1164ff9700 1 -- 172.21.15.103:0/3186940527 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f115c0070b0 con 0x7f116812f250 2023-07-02T09:03:37.484 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.481+0000 7f1164ff9700 1 --2- 172.21.15.103:0/3186940527 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11540377c0 0x7f1154039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:37.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.481+0000 7f1164ff9700 1 -- 172.21.15.103:0/3186940527 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f115c046b20 con 0x7f116812f250 2023-07-02T09:03:37.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.481+0000 7f11677fe700 1 --2- 172.21.15.103:0/3186940527 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11540377c0 0x7f1154039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:37.485 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.482+0000 7f11677fe700 1 --2- 172.21.15.103:0/3186940527 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11540377c0 0x7f1154039cd0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:37.486 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.486+0000 7f1164ff9700 1 -- 172.21.15.103:0/3186940527 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f115c00fdf0 con 0x7f116812f250 2023-07-02T09:03:37.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.671+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1168073ca0 con 0x7f116812f250 2023-07-02T09:03:37.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.671+0000 7f1164ff9700 1 -- 172.21.15.103:0/3186940527 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f115c00ddd0 con 0x7f116812f250 2023-07-02T09:03:37.672 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:37.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11540377c0 msgr2=0x7f1154039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:37.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 --2- 172.21.15.103:0/3186940527 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11540377c0 0x7f1154039cd0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:37.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f116812f250 msgr2=0x7f11681c3ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:37.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 --2- 172.21.15.103:0/3186940527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f116812f250 0x7f11681c3ac0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:37.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 shutdown_connections 2023-07-02T09:03:37.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 --2- 172.21.15.103:0/3186940527 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f11540377c0 0x7f1154039cd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:37.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 --2- 172.21.15.103:0/3186940527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f116812f250 0x7f11681c3ac0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:37.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 >> 172.21.15.103:0/3186940527 conn(0x7f116809d810 msgr2=0x7f116812e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:37.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.673+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 shutdown_connections 2023-07-02T09:03:37.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:37.674+0000 7f116e854700 1 -- 172.21.15.103:0/3186940527 wait complete. 2023-07-02T09:03:37.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:38.687 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:38.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.828+0000 7f67f6b70700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:38.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.833+0000 7f67f6b70700 1 Processor -- start 2023-07-02T09:03:38.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.833+0000 7f67f6b70700 1 -- start start 2023-07-02T09:03:38.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.833+0000 7f67f6b70700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f67f012f250 0x7f67f012f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:38.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67f6b70700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f006df60 con 0x7f67f0098a00 2023-07-02T09:03:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67f6b70700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f009a710 con 0x7f67f012f250 2023-07-02T09:03:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67f490c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f67f012f250 0x7f67f012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67effff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f67f0098a00 0x7f67f0098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37272/0 (socket says 172.21.15.103:37272) 2023-07-02T09:03:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67effff700 1 -- 172.21.15.103:0/2575696398 learned_addr learned my addr 172.21.15.103:0/2575696398 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67f490c700 1 -- 172.21.15.103:0/2575696398 >> v1:172.21.15.103:6789/0 conn(0x7f67f0098a00 legacy=0x7f67f0098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67ef7fe700 1 -- 172.21.15.103:0/2575696398 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748481672 0 0) 0x7f67f006df60 con 0x7f67f0098a00 2023-07-02T09:03:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67f490c700 1 -- 172.21.15.103:0/2575696398 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f012ed20 con 0x7f67f012f250 2023-07-02T09:03:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67f490c700 1 --2- 172.21.15.103:0/2575696398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f012f250 0x7f67f012f660 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ba957e167280aa server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:38.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67ef7fe700 1 -- 172.21.15.103:0/2575696398 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67e4009cd0 con 0x7f67f012f250 2023-07-02T09:03:38.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67ef7fe700 1 -- 172.21.15.103:0/2575696398 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f67e4009e30 con 0x7f67f012f250 2023-07-02T09:03:38.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.834+0000 7f67ef7fe700 1 -- 172.21.15.103:0/2575696398 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67e400e360 con 0x7f67f012f250 2023-07-02T09:03:38.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.835+0000 7f67f6b70700 1 -- 172.21.15.103:0/2575696398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f012f250 msgr2=0x7f67f012f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:38.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.835+0000 7f67f6b70700 1 --2- 172.21.15.103:0/2575696398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f012f250 0x7f67f012f660 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:38.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.835+0000 7f67f6b70700 1 -- 172.21.15.103:0/2575696398 shutdown_connections 2023-07-02T09:03:38.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.835+0000 7f67f6b70700 1 --2- 172.21.15.103:0/2575696398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f012f250 0x7f67f012f660 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:38.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.835+0000 7f67f6b70700 1 -- 172.21.15.103:0/2575696398 >> 172.21.15.103:0/2575696398 conn(0x7f67f009d810 msgr2=0x7f67f009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:38.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.835+0000 7f67f6b70700 1 -- 172.21.15.103:0/2575696398 shutdown_connections 2023-07-02T09:03:38.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.835+0000 7f67f6b70700 1 -- 172.21.15.103:0/2575696398 wait complete. 2023-07-02T09:03:38.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.835+0000 7f67f6b70700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:38.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.836+0000 7f67f6b70700 1 Processor -- start 2023-07-02T09:03:38.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.836+0000 7f67f6b70700 1 -- start start 2023-07-02T09:03:38.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.837+0000 7f67f6b70700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f0098a00 0x7f67f0117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.837+0000 7f67f6b70700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f011a390 con 0x7f67f0098a00 2023-07-02T09:03:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67f490c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f0098a00 0x7f67f0117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67f490c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f0098a00 0x7f67f0117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58266/0 (socket says 172.21.15.103:58266) 2023-07-02T09:03:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67f490c700 1 -- 172.21.15.103:0/2811916442 learned_addr learned my addr 172.21.15.103:0/2811916442 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67f490c700 1 -- 172.21.15.103:0/2811916442 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f0099b60 con 0x7f67f0098a00 2023-07-02T09:03:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67f490c700 1 --2- 172.21.15.103:0/2811916442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f0098a00 0x7f67f0117360 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67ed7fa700 1 -- 172.21.15.103:0/2811916442 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67e4014540 con 0x7f67f0098a00 2023-07-02T09:03:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67ed7fa700 1 -- 172.21.15.103:0/2811916442 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f67e40146a0 con 0x7f67f0098a00 2023-07-02T09:03:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f01196f0 con 0x7f67f0098a00 2023-07-02T09:03:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67ed7fa700 1 -- 172.21.15.103:0/2811916442 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67e4014940 con 0x7f67f0098a00 2023-07-02T09:03:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.838+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f0119db0 con 0x7f67f0098a00 2023-07-02T09:03:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.839+0000 7f67ed7fa700 1 -- 172.21.15.103:0/2811916442 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f67e4014af0 con 0x7f67f0098a00 2023-07-02T09:03:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.839+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f0061ea0 con 0x7f67f0098a00 2023-07-02T09:03:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.839+0000 7f67ed7fa700 1 --2- 172.21.15.103:0/2811916442 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67d40377b0 0x7f67d4039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.839+0000 7f67ed7fa700 1 -- 172.21.15.103:0/2811916442 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f67e4048200 con 0x7f67f0098a00 2023-07-02T09:03:38.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.839+0000 7f67effff700 1 --2- 172.21.15.103:0/2811916442 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67d40377b0 0x7f67d4039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:38.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.840+0000 7f67effff700 1 --2- 172.21.15.103:0/2811916442 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67d40377b0 0x7f67d4039cc0 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:38.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:38.844+0000 7f67ed7fa700 1 -- 172.21.15.103:0/2811916442 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f67e400fb00 con 0x7f67f0098a00 2023-07-02T09:03:39.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.033+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f67f0099b60 con 0x7f67f0098a00 2023-07-02T09:03:39.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.033+0000 7f67ed7fa700 1 -- 172.21.15.103:0/2811916442 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f67f0099b60 con 0x7f67f0098a00 2023-07-02T09:03:39.034 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:39.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67d40377b0 msgr2=0x7f67d4039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:39.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 --2- 172.21.15.103:0/2811916442 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67d40377b0 0x7f67d4039cc0 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:39.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f0098a00 msgr2=0x7f67f0117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:39.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 --2- 172.21.15.103:0/2811916442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f0098a00 0x7f67f0117360 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:39.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 shutdown_connections 2023-07-02T09:03:39.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 --2- 172.21.15.103:0/2811916442 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f67d40377b0 0x7f67d4039cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:39.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 --2- 172.21.15.103:0/2811916442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67f0098a00 0x7f67f0117360 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:39.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 >> 172.21.15.103:0/2811916442 conn(0x7f67f009d810 msgr2=0x7f67f00a0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:39.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.035+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 shutdown_connections 2023-07-02T09:03:39.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:39.036+0000 7f67f6b70700 1 -- 172.21.15.103:0/2811916442 wait complete. 2023-07-02T09:03:39.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:40.049 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:40.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.189+0000 7f6b3b97a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:40.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.194+0000 7f6b3b97a700 1 Processor -- start 2023-07-02T09:03:40.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.194+0000 7f6b3b97a700 1 -- start start 2023-07-02T09:03:40.195 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b3b97a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6b34133360 0x7f6b3409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b3b97a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3409e260 con 0x7f6b34132f50 2023-07-02T09:03:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b3b97a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b340a1640 con 0x7f6b34133360 2023-07-02T09:03:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b38f15700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6b34133360 0x7f6b3409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b39716700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6b34132f50 0x7f6b3409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37280/0 (socket says 172.21.15.103:37280) 2023-07-02T09:03:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b39716700 1 -- 172.21.15.103:0/1777279234 learned_addr learned my addr 172.21.15.103:0/1777279234 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b38f15700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6b34133360 0x7f6b3409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58274/0 (socket says 172.21.15.103:58274) 2023-07-02T09:03:40.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b38f15700 1 -- 172.21.15.103:0/1777279234 >> v1:172.21.15.103:6789/0 conn(0x7f6b34132f50 legacy=0x7f6b3409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:40.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b38f15700 1 -- 172.21.15.103:0/1777279234 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b34132a20 con 0x7f6b34133360 2023-07-02T09:03:40.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b38f15700 1 --2- 172.21.15.103:0/1777279234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34133360 0x7f6b3409cc90 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc063d402b52615 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:40.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.195+0000 7f6b2bfff700 1 -- 172.21.15.103:0/1777279234 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6b30009cd0 con 0x7f6b34133360 2023-07-02T09:03:40.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b2bfff700 1 -- 172.21.15.103:0/1777279234 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6b30009e30 con 0x7f6b34133360 2023-07-02T09:03:40.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b2bfff700 1 -- 172.21.15.103:0/1777279234 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6b3000e360 con 0x7f6b34133360 2023-07-02T09:03:40.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1777279234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34133360 msgr2=0x7f6b3409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:40.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b3b97a700 1 --2- 172.21.15.103:0/1777279234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34133360 0x7f6b3409cc90 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:40.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1777279234 shutdown_connections 2023-07-02T09:03:40.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b3b97a700 1 --2- 172.21.15.103:0/1777279234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34133360 0x7f6b3409cc90 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:40.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1777279234 >> 172.21.15.103:0/1777279234 conn(0x7f6b340a2080 msgr2=0x7f6b340a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:40.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1777279234 shutdown_connections 2023-07-02T09:03:40.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.196+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1777279234 wait complete. 2023-07-02T09:03:40.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.197+0000 7f6b3b97a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.197+0000 7f6b3b97a700 1 Processor -- start 2023-07-02T09:03:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.197+0000 7f6b3b97a700 1 -- start start 2023-07-02T09:03:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.198+0000 7f6b3b97a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34132f50 0x7f6b3412bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.198+0000 7f6b3b97a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3406d260 con 0x7f6b34132f50 2023-07-02T09:03:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.198+0000 7f6b39716700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34132f50 0x7f6b3412bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.198+0000 7f6b39716700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34132f50 0x7f6b3412bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58284/0 (socket says 172.21.15.103:58284) 2023-07-02T09:03:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.198+0000 7f6b39716700 1 -- 172.21.15.103:0/1462562278 learned_addr learned my addr 172.21.15.103:0/1462562278 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:40.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.199+0000 7f6b39716700 1 -- 172.21.15.103:0/1462562278 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3412bb90 con 0x7f6b34132f50 2023-07-02T09:03:40.202 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.199+0000 7f6b39716700 1 --2- 172.21.15.103:0/1462562278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34132f50 0x7f6b3412bea0 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.199+0000 7f6b29ffb700 1 -- 172.21.15.103:0/1462562278 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6b24005a60 con 0x7f6b34132f50 2023-07-02T09:03:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.199+0000 7f6b29ffb700 1 -- 172.21.15.103:0/1462562278 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6b24005bc0 con 0x7f6b34132f50 2023-07-02T09:03:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.199+0000 7f6b29ffb700 1 -- 172.21.15.103:0/1462562278 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6b24005e60 con 0x7f6b34132f50 2023-07-02T09:03:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.199+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3406f730 con 0x7f6b34132f50 2023-07-02T09:03:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.199+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3406f490 con 0x7f6b34132f50 2023-07-02T09:03:40.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.200+0000 7f6b29ffb700 1 -- 172.21.15.103:0/1462562278 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6b240070b0 con 0x7f6b34132f50 2023-07-02T09:03:40.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.200+0000 7f6b29ffb700 1 --2- 172.21.15.103:0/1462562278 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6b180377c0 0x7f6b18039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:40.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.200+0000 7f6b29ffb700 1 -- 172.21.15.103:0/1462562278 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6b24046ab0 con 0x7f6b34132f50 2023-07-02T09:03:40.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.200+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3406f490 con 0x7f6b34132f50 2023-07-02T09:03:40.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.200+0000 7f6b38f15700 1 --2- 172.21.15.103:0/1462562278 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6b180377c0 0x7f6b18039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:40.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.201+0000 7f6b38f15700 1 --2- 172.21.15.103:0/1462562278 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6b180377c0 0x7f6b18039cd0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:40.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.205+0000 7f6b29ffb700 1 -- 172.21.15.103:0/1462562278 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6b3406f490 con 0x7f6b34132f50 2023-07-02T09:03:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.391+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6b3406f490 con 0x7f6b34132f50 2023-07-02T09:03:40.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.391+0000 7f6b29ffb700 1 -- 172.21.15.103:0/1462562278 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6b3406f490 con 0x7f6b34132f50 2023-07-02T09:03:40.392 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6b180377c0 msgr2=0x7f6b18039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 --2- 172.21.15.103:0/1462562278 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6b180377c0 0x7f6b18039cd0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34132f50 msgr2=0x7f6b3412bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 --2- 172.21.15.103:0/1462562278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34132f50 0x7f6b3412bea0 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 shutdown_connections 2023-07-02T09:03:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 --2- 172.21.15.103:0/1462562278 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f6b180377c0 0x7f6b18039cd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:40.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 --2- 172.21.15.103:0/1462562278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b34132f50 0x7f6b3412bea0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:40.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 >> 172.21.15.103:0/1462562278 conn(0x7f6b340a2080 msgr2=0x7f6b341bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:40.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 shutdown_connections 2023-07-02T09:03:40.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:40.393+0000 7f6b3b97a700 1 -- 172.21.15.103:0/1462562278 wait complete. 2023-07-02T09:03:40.406 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:41.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:41.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.545+0000 7f25b4844700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:41.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.550+0000 7f25b4844700 1 Processor -- start 2023-07-02T09:03:41.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.550+0000 7f25b4844700 1 -- start start 2023-07-02T09:03:41.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25b4844700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f25b0133360 0x7f25b009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:41.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25b4844700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b009e260 con 0x7f25b0132f50 2023-07-02T09:03:41.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25b4844700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b00a1640 con 0x7f25b0133360 2023-07-02T09:03:41.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25add9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f25b0133360 0x7f25b009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:41.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25ae59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f25b0132f50 0x7f25b009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37288/0 (socket says 172.21.15.103:37288) 2023-07-02T09:03:41.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25add9b700 1 --2- 172.21.15.103:0/2523072911 >> v2:172.21.15.103:3300/0 conn(0x7f25b0133360 0x7f25b009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58290/0 (socket says 172.21.15.103:58290) 2023-07-02T09:03:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25ae59c700 1 -- 172.21.15.103:0/2523072911 learned_addr learned my addr 172.21.15.103:0/2523072911 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25add9b700 1 -- 172.21.15.103:0/2523072911 >> v1:172.21.15.103:6789/0 conn(0x7f25b0132f50 legacy=0x7f25b009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25add9b700 1 -- 172.21.15.103:0/2523072911 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b0132a20 con 0x7f25b0133360 2023-07-02T09:03:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25ad59a700 1 -- 172.21.15.103:0/2523072911 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598817290 0 0) 0x7f25b009e260 con 0x7f25b0132f50 2023-07-02T09:03:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25add9b700 1 --2- 172.21.15.103:0/2523072911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0133360 0x7f25b009cc90 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d9603a8514666e server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.551+0000 7f25ad59a700 1 -- 172.21.15.103:0/2523072911 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f25a0009cd0 con 0x7f25b0133360 2023-07-02T09:03:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25ad59a700 1 -- 172.21.15.103:0/2523072911 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f25a0009e30 con 0x7f25b0133360 2023-07-02T09:03:41.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25ad59a700 1 -- 172.21.15.103:0/2523072911 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f25a000e360 con 0x7f25b0133360 2023-07-02T09:03:41.555 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25b4844700 1 -- 172.21.15.103:0/2523072911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0133360 msgr2=0x7f25b009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:41.556 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25b4844700 1 --2- 172.21.15.103:0/2523072911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0133360 0x7f25b009cc90 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:41.556 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25b4844700 1 -- 172.21.15.103:0/2523072911 shutdown_connections 2023-07-02T09:03:41.556 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25b4844700 1 --2- 172.21.15.103:0/2523072911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0133360 0x7f25b009cc90 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:41.556 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25b4844700 1 -- 172.21.15.103:0/2523072911 >> 172.21.15.103:0/2523072911 conn(0x7f25b00a2080 msgr2=0x7f25b00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:41.556 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25b4844700 1 -- 172.21.15.103:0/2523072911 shutdown_connections 2023-07-02T09:03:41.556 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.552+0000 7f25b4844700 1 -- 172.21.15.103:0/2523072911 wait complete. 2023-07-02T09:03:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.553+0000 7f25b4844700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.554+0000 7f25b4844700 1 Processor -- start 2023-07-02T09:03:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.554+0000 7f25b4844700 1 -- start start 2023-07-02T09:03:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25b4844700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0132f50 0x7f25b012c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25b4844700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b006d2f0 con 0x7f25b0132f50 2023-07-02T09:03:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25ae59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0132f50 0x7f25b012c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25ae59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0132f50 0x7f25b012c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58306/0 (socket says 172.21.15.103:58306) 2023-07-02T09:03:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25ae59c700 1 -- 172.21.15.103:0/701163612 learned_addr learned my addr 172.21.15.103:0/701163612 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25ae59c700 1 -- 172.21.15.103:0/701163612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b006dd60 con 0x7f25b0132f50 2023-07-02T09:03:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25ae59c700 1 --2- 172.21.15.103:0/701163612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0132f50 0x7f25b012c1a0 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f259effd700 1 -- 172.21.15.103:0/701163612 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f25a4004b80 con 0x7f25b0132f50 2023-07-02T09:03:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f259effd700 1 -- 172.21.15.103:0/701163612 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f25a4004ce0 con 0x7f25b0132f50 2023-07-02T09:03:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f259effd700 1 -- 172.21.15.103:0/701163612 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f25a400c4f0 con 0x7f25b0132f50 2023-07-02T09:03:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b012bc50 con 0x7f25b0132f50 2023-07-02T09:03:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.555+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b006f850 con 0x7f25b0132f50 2023-07-02T09:03:41.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.556+0000 7f259effd700 1 -- 172.21.15.103:0/701163612 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f25a400c650 con 0x7f25b0132f50 2023-07-02T09:03:41.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.556+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b006dbb0 con 0x7f25b0132f50 2023-07-02T09:03:41.559 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.556+0000 7f259effd700 1 --2- 172.21.15.103:0/701163612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f25900377c0 0x7f2590039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:41.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.556+0000 7f259effd700 1 -- 172.21.15.103:0/701163612 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f25a4046970 con 0x7f25b0132f50 2023-07-02T09:03:41.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.557+0000 7f25add9b700 1 --2- 172.21.15.103:0/701163612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f25900377c0 0x7f2590039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:41.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.557+0000 7f25add9b700 1 --2- 172.21.15.103:0/701163612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f25900377c0 0x7f2590039cd0 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:41.561 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.561+0000 7f259effd700 1 -- 172.21.15.103:0/701163612 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f25a4009c80 con 0x7f25b0132f50 2023-07-02T09:03:41.746 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.746+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f25b0061ea0 con 0x7f25b0132f50 2023-07-02T09:03:41.747 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.746+0000 7f259effd700 1 -- 172.21.15.103:0/701163612 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f25a40098a0 con 0x7f25b0132f50 2023-07-02T09:03:41.747 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f25900377c0 msgr2=0x7f2590039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 --2- 172.21.15.103:0/701163612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f25900377c0 0x7f2590039cd0 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0132f50 msgr2=0x7f25b012c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 --2- 172.21.15.103:0/701163612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0132f50 0x7f25b012c1a0 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 shutdown_connections 2023-07-02T09:03:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 --2- 172.21.15.103:0/701163612 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f25900377c0 0x7f2590039cd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 --2- 172.21.15.103:0/701163612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b0132f50 0x7f25b012c1a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 >> 172.21.15.103:0/701163612 conn(0x7f25b00a2080 msgr2=0x7f25b0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:41.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 shutdown_connections 2023-07-02T09:03:41.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:41.748+0000 7f25b4844700 1 -- 172.21.15.103:0/701163612 wait complete. 2023-07-02T09:03:41.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:42.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~237s 2023-07-02T09:03:42.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:42.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.901+0000 7ff904cca700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:42.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.906+0000 7ff904cca700 1 Processor -- start 2023-07-02T09:03:42.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.906+0000 7ff904cca700 1 -- start start 2023-07-02T09:03:42.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff904cca700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff900132030 0x7ff900134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:42.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff904cca700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90006df60 con 0x7ff90012f250 2023-07-02T09:03:42.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff904cca700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90009a710 con 0x7ff900132030 2023-07-02T09:03:42.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fdd9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff900132030 0x7ff900134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:42.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fe59c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff90012f250 0x7ff90012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:37292/0 (socket says 172.21.15.103:37292) 2023-07-02T09:03:42.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fe59c700 1 -- 172.21.15.103:0/2669479587 learned_addr learned my addr 172.21.15.103:0/2669479587 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:42.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fdd9b700 1 -- 172.21.15.103:0/2669479587 >> v1:172.21.15.103:6789/0 conn(0x7ff90012f250 legacy=0x7ff90012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:42.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fdd9b700 1 -- 172.21.15.103:0/2669479587 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90012ed20 con 0x7ff900132030 2023-07-02T09:03:42.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fdd9b700 1 --2- 172.21.15.103:0/2669479587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff900132030 0x7ff900134450 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324413885dad35d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:42.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fd59a700 1 -- 172.21.15.103:0/2669479587 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff8f4009cd0 con 0x7ff900132030 2023-07-02T09:03:42.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fd59a700 1 -- 172.21.15.103:0/2669479587 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff8f4009e30 con 0x7ff900132030 2023-07-02T09:03:42.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.907+0000 7ff8fd59a700 1 -- 172.21.15.103:0/2669479587 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff8f400f470 con 0x7ff900132030 2023-07-02T09:03:42.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.908+0000 7ff904cca700 1 -- 172.21.15.103:0/2669479587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff900132030 msgr2=0x7ff900134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:42.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.908+0000 7ff904cca700 1 --2- 172.21.15.103:0/2669479587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff900132030 0x7ff900134450 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:42.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.908+0000 7ff904cca700 1 -- 172.21.15.103:0/2669479587 shutdown_connections 2023-07-02T09:03:42.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.908+0000 7ff904cca700 1 --2- 172.21.15.103:0/2669479587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff900132030 0x7ff900134450 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:42.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.908+0000 7ff904cca700 1 -- 172.21.15.103:0/2669479587 >> 172.21.15.103:0/2669479587 conn(0x7ff90009d810 msgr2=0x7ff90009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:42.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.908+0000 7ff904cca700 1 -- 172.21.15.103:0/2669479587 shutdown_connections 2023-07-02T09:03:42.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.908+0000 7ff904cca700 1 -- 172.21.15.103:0/2669479587 wait complete. 2023-07-02T09:03:42.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.908+0000 7ff904cca700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:42.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.910+0000 7ff904cca700 1 Processor -- start 2023-07-02T09:03:42.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.910+0000 7ff904cca700 1 -- start start 2023-07-02T09:03:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.910+0000 7ff904cca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff90012f250 0x7ff9001c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.910+0000 7ff904cca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90011b9d0 con 0x7ff90012f250 2023-07-02T09:03:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.910+0000 7ff8fe59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff90012f250 0x7ff9001c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.910+0000 7ff8fe59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff90012f250 0x7ff9001c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58316/0 (socket says 172.21.15.103:58316) 2023-07-02T09:03:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.910+0000 7ff8fe59c700 1 -- 172.21.15.103:0/2397752880 learned_addr learned my addr 172.21.15.103:0/2397752880 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.911+0000 7ff8fe59c700 1 -- 172.21.15.103:0/2397752880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff900071340 con 0x7ff90012f250 2023-07-02T09:03:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.911+0000 7ff8fe59c700 1 --2- 172.21.15.103:0/2397752880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff90012f250 0x7ff9001c7250 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:42.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.911+0000 7ff8eeffd700 1 -- 172.21.15.103:0/2397752880 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff8f0005a60 con 0x7ff90012f250 2023-07-02T09:03:42.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.911+0000 7ff8eeffd700 1 -- 172.21.15.103:0/2397752880 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff8f0005bc0 con 0x7ff90012f250 2023-07-02T09:03:42.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.911+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff900071d60 con 0x7ff90012f250 2023-07-02T09:03:42.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.911+0000 7ff8eeffd700 1 -- 172.21.15.103:0/2397752880 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff8f0005e60 con 0x7ff90012f250 2023-07-02T09:03:42.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.911+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff900129060 con 0x7ff90012f250 2023-07-02T09:03:42.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.912+0000 7ff8eeffd700 1 -- 172.21.15.103:0/2397752880 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ff8f00070b0 con 0x7ff90012f250 2023-07-02T09:03:42.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.912+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90011bb10 con 0x7ff90012f250 2023-07-02T09:03:42.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.912+0000 7ff8eeffd700 1 --2- 172.21.15.103:0/2397752880 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8e00377c0 0x7ff8e0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:42.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.912+0000 7ff8eeffd700 1 -- 172.21.15.103:0/2397752880 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ff8f0046bf0 con 0x7ff90012f250 2023-07-02T09:03:42.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.912+0000 7ff8fdd9b700 1 --2- 172.21.15.103:0/2397752880 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8e00377c0 0x7ff8e0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:42.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.913+0000 7ff8fdd9b700 1 --2- 172.21.15.103:0/2397752880 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8e00377c0 0x7ff8e0039cd0 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:42.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:42.917+0000 7ff8eeffd700 1 -- 172.21.15.103:0/2397752880 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ff8f000fdf0 con 0x7ff90012f250 2023-07-02T09:03:43.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.104+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff900061ea0 con 0x7ff90012f250 2023-07-02T09:03:43.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.104+0000 7ff8eeffd700 1 -- 172.21.15.103:0/2397752880 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ff8f000ddd0 con 0x7ff90012f250 2023-07-02T09:03:43.105 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:43.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8e00377c0 msgr2=0x7ff8e0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:43.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 --2- 172.21.15.103:0/2397752880 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8e00377c0 0x7ff8e0039cd0 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:43.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff90012f250 msgr2=0x7ff9001c7250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:43.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 --2- 172.21.15.103:0/2397752880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff90012f250 0x7ff9001c7250 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:43.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 shutdown_connections 2023-07-02T09:03:43.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 --2- 172.21.15.103:0/2397752880 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff8e00377c0 0x7ff8e0039cd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:43.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 --2- 172.21.15.103:0/2397752880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff90012f250 0x7ff9001c7250 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:43.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 >> 172.21.15.103:0/2397752880 conn(0x7ff90009d810 msgr2=0x7ff90012e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:43.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 shutdown_connections 2023-07-02T09:03:43.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:43.106+0000 7ff904cca700 1 -- 172.21.15.103:0/2397752880 wait complete. 2023-07-02T09:03:43.119 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:44.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.258+0000 7f44ef13d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.263+0000 7f44ef13d700 1 Processor -- start 2023-07-02T09:03:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.263+0000 7f44ef13d700 1 -- start start 2023-07-02T09:03:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.264+0000 7f44ef13d700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44e8133360 0x7f44e809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.264+0000 7f44ef13d700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e809e260 con 0x7f44e8132f50 2023-07-02T09:03:44.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.264+0000 7f44ef13d700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e80a1640 con 0x7f44e8133360 2023-07-02T09:03:44.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.264+0000 7f44e7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44e8133360 0x7f44e809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:44.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.264+0000 7f44eced9700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f44e8132f50 0x7f44e809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40976/0 (socket says 172.21.15.103:40976) 2023-07-02T09:03:44.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.264+0000 7f44e7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44e8133360 0x7f44e809cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48392/0 (socket says 172.21.15.103:48392) 2023-07-02T09:03:44.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.264+0000 7f44eced9700 1 -- 172.21.15.103:0/1104242268 learned_addr learned my addr 172.21.15.103:0/1104242268 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:44.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.265+0000 7f44e7fff700 1 -- 172.21.15.103:0/1104242268 >> v1:172.21.15.103:6789/0 conn(0x7f44e8132f50 legacy=0x7f44e809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.265+0000 7f44e7fff700 1 -- 172.21.15.103:0/1104242268 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e8132a20 con 0x7f44e8133360 2023-07-02T09:03:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.265+0000 7f44e7fff700 1 --2- 172.21.15.103:0/1104242268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8133360 0x7f44e809cc90 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ef1b30ea0b04ef server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.265+0000 7f44e77fe700 1 -- 172.21.15.103:0/1104242268 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f44d4009cd0 con 0x7f44e8133360 2023-07-02T09:03:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.265+0000 7f44e77fe700 1 -- 172.21.15.103:0/1104242268 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f44d4009e30 con 0x7f44e8133360 2023-07-02T09:03:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.265+0000 7f44e77fe700 1 -- 172.21.15.103:0/1104242268 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f44d400e360 con 0x7f44e8133360 2023-07-02T09:03:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.265+0000 7f44ef13d700 1 -- 172.21.15.103:0/1104242268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8133360 msgr2=0x7f44e809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:44.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.265+0000 7f44ef13d700 1 --2- 172.21.15.103:0/1104242268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8133360 0x7f44e809cc90 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.266+0000 7f44ef13d700 1 -- 172.21.15.103:0/1104242268 shutdown_connections 2023-07-02T09:03:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.266+0000 7f44ef13d700 1 --2- 172.21.15.103:0/1104242268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8133360 0x7f44e809cc90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.266+0000 7f44ef13d700 1 -- 172.21.15.103:0/1104242268 >> 172.21.15.103:0/1104242268 conn(0x7f44e80a2080 msgr2=0x7f44e80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.266+0000 7f44ef13d700 1 -- 172.21.15.103:0/1104242268 shutdown_connections 2023-07-02T09:03:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.266+0000 7f44ef13d700 1 -- 172.21.15.103:0/1104242268 wait complete. 2023-07-02T09:03:44.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.266+0000 7f44ef13d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:44.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.267+0000 7f44ef13d700 1 Processor -- start 2023-07-02T09:03:44.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.267+0000 7f44ef13d700 1 -- start start 2023-07-02T09:03:44.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44ef13d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8132f50 0x7f44e812bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:44.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44ef13d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e806d260 con 0x7f44e8132f50 2023-07-02T09:03:44.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44eced9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8132f50 0x7f44e812bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:44.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44eced9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8132f50 0x7f44e812bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48396/0 (socket says 172.21.15.103:48396) 2023-07-02T09:03:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44eced9700 1 -- 172.21.15.103:0/1212468486 learned_addr learned my addr 172.21.15.103:0/1212468486 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44eced9700 1 -- 172.21.15.103:0/1212468486 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e812bb90 con 0x7f44e8132f50 2023-07-02T09:03:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44eced9700 1 --2- 172.21.15.103:0/1212468486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8132f50 0x7f44e812bea0 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44e57fa700 1 -- 172.21.15.103:0/1212468486 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f44dc005a60 con 0x7f44e8132f50 2023-07-02T09:03:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44e57fa700 1 -- 172.21.15.103:0/1212468486 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f44dc005bc0 con 0x7f44e8132f50 2023-07-02T09:03:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e806f730 con 0x7f44e8132f50 2023-07-02T09:03:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44e57fa700 1 -- 172.21.15.103:0/1212468486 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f44dc005e60 con 0x7f44e8132f50 2023-07-02T09:03:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.268+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e806f490 con 0x7f44e8132f50 2023-07-02T09:03:44.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.269+0000 7f44e57fa700 1 -- 172.21.15.103:0/1212468486 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f44dc0070b0 con 0x7f44e8132f50 2023-07-02T09:03:44.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.269+0000 7f44e57fa700 1 --2- 172.21.15.103:0/1212468486 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f44cc0377c0 0x7f44cc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:44.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.269+0000 7f44e57fa700 1 -- 172.21.15.103:0/1212468486 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f44dc046ab0 con 0x7f44e8132f50 2023-07-02T09:03:44.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.269+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e806f490 con 0x7f44e8132f50 2023-07-02T09:03:44.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.270+0000 7f44e7fff700 1 --2- 172.21.15.103:0/1212468486 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f44cc0377c0 0x7f44cc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:44.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.270+0000 7f44e7fff700 1 --2- 172.21.15.103:0/1212468486 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f44cc0377c0 0x7f44cc039cd0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:44.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.274+0000 7f44e57fa700 1 -- 172.21.15.103:0/1212468486 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f44e806f490 con 0x7f44e8132f50 2023-07-02T09:03:44.461 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.461+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f44e806f490 con 0x7f44e8132f50 2023-07-02T09:03:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.462+0000 7f44e57fa700 1 -- 172.21.15.103:0/1212468486 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f44e806f490 con 0x7f44e8132f50 2023-07-02T09:03:44.462 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f44cc0377c0 msgr2=0x7f44cc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 --2- 172.21.15.103:0/1212468486 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f44cc0377c0 0x7f44cc039cd0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8132f50 msgr2=0x7f44e812bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:44.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 --2- 172.21.15.103:0/1212468486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8132f50 0x7f44e812bea0 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:44.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 shutdown_connections 2023-07-02T09:03:44.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 --2- 172.21.15.103:0/1212468486 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f44cc0377c0 0x7f44cc039cd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:44.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 --2- 172.21.15.103:0/1212468486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44e8132f50 0x7f44e812bea0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:44.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 >> 172.21.15.103:0/1212468486 conn(0x7f44e80a2080 msgr2=0x7f44e81bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:44.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 shutdown_connections 2023-07-02T09:03:44.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:44.464+0000 7f44ef13d700 1 -- 172.21.15.103:0/1212468486 wait complete. 2023-07-02T09:03:44.476 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:45.477 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:45.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.609+0000 7f1fc76f6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:45.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.614+0000 7f1fc76f6700 1 Processor -- start 2023-07-02T09:03:45.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.614+0000 7f1fc76f6700 1 -- start start 2023-07-02T09:03:45.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc76f6700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fc0133360 0x7f1fc009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:45.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc76f6700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc009e260 con 0x7f1fc0132f50 2023-07-02T09:03:45.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc76f6700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc00a1640 con 0x7f1fc0133360 2023-07-02T09:03:45.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc4c91700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fc0133360 0x7f1fc009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:45.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc5492700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1fc0132f50 0x7f1fc009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40984/0 (socket says 172.21.15.103:40984) 2023-07-02T09:03:45.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc4c91700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fc0133360 0x7f1fc009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48410/0 (socket says 172.21.15.103:48410) 2023-07-02T09:03:45.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc5492700 1 -- 172.21.15.103:0/3998799392 learned_addr learned my addr 172.21.15.103:0/3998799392 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:45.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc4c91700 1 -- 172.21.15.103:0/3998799392 >> v1:172.21.15.103:6789/0 conn(0x7f1fc0132f50 legacy=0x7f1fc009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:45.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fb7fff700 1 -- 172.21.15.103:0/3998799392 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640468596 0 0) 0x7f1fc009e260 con 0x7f1fc0132f50 2023-07-02T09:03:45.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc4c91700 1 -- 172.21.15.103:0/3998799392 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc0132a20 con 0x7f1fc0133360 2023-07-02T09:03:45.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fc4c91700 1 --2- 172.21.15.103:0/3998799392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0133360 0x7f1fc009cc90 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f1ddd3e66df478 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:45.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fb7fff700 1 -- 172.21.15.103:0/3998799392 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1fb0009cd0 con 0x7f1fc0133360 2023-07-02T09:03:45.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fb7fff700 1 -- 172.21.15.103:0/3998799392 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1fb0009e30 con 0x7f1fc0133360 2023-07-02T09:03:45.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.615+0000 7f1fb7fff700 1 -- 172.21.15.103:0/3998799392 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1fb000e360 con 0x7f1fc0133360 2023-07-02T09:03:45.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.616+0000 7f1fc76f6700 1 -- 172.21.15.103:0/3998799392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0133360 msgr2=0x7f1fc009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:45.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.616+0000 7f1fc76f6700 1 --2- 172.21.15.103:0/3998799392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0133360 0x7f1fc009cc90 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:45.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.616+0000 7f1fc76f6700 1 -- 172.21.15.103:0/3998799392 shutdown_connections 2023-07-02T09:03:45.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.616+0000 7f1fc76f6700 1 --2- 172.21.15.103:0/3998799392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0133360 0x7f1fc009cc90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:45.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.616+0000 7f1fc76f6700 1 -- 172.21.15.103:0/3998799392 >> 172.21.15.103:0/3998799392 conn(0x7f1fc00a2080 msgr2=0x7f1fc00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:45.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.616+0000 7f1fc76f6700 1 -- 172.21.15.103:0/3998799392 shutdown_connections 2023-07-02T09:03:45.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.616+0000 7f1fc76f6700 1 -- 172.21.15.103:0/3998799392 wait complete. 2023-07-02T09:03:45.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.616+0000 7f1fc76f6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:45.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.617+0000 7f1fc76f6700 1 Processor -- start 2023-07-02T09:03:45.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.617+0000 7f1fc76f6700 1 -- start start 2023-07-02T09:03:45.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fc76f6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0132f50 0x7f1fc01ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:45.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fc76f6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc006d370 con 0x7f1fc0132f50 2023-07-02T09:03:45.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fc5492700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0132f50 0x7f1fc01ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:45.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fc5492700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0132f50 0x7f1fc01ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48418/0 (socket says 172.21.15.103:48418) 2023-07-02T09:03:45.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fc5492700 1 -- 172.21.15.103:0/4167450575 learned_addr learned my addr 172.21.15.103:0/4167450575 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:45.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fc5492700 1 -- 172.21.15.103:0/4167450575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc006dde0 con 0x7f1fc0132f50 2023-07-02T09:03:45.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fc5492700 1 --2- 172.21.15.103:0/4167450575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0132f50 0x7f1fc01ba9b0 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:45.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fb5ffb700 1 -- 172.21.15.103:0/4167450575 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1fbc004b80 con 0x7f1fc0132f50 2023-07-02T09:03:45.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.618+0000 7f1fb5ffb700 1 -- 172.21.15.103:0/4167450575 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1fbc004ce0 con 0x7f1fc0132f50 2023-07-02T09:03:45.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.619+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc01ba4f0 con 0x7f1fc0132f50 2023-07-02T09:03:45.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.619+0000 7f1fb5ffb700 1 -- 172.21.15.103:0/4167450575 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1fbc00c4f0 con 0x7f1fc0132f50 2023-07-02T09:03:45.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.619+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc0071e40 con 0x7f1fc0132f50 2023-07-02T09:03:45.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.619+0000 7f1fb5ffb700 1 -- 172.21.15.103:0/4167450575 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f1fbc00c650 con 0x7f1fc0132f50 2023-07-02T09:03:45.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.620+0000 7f1fb5ffb700 1 --2- 172.21.15.103:0/4167450575 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1fa40377c0 0x7f1fa4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:45.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.620+0000 7f1fb5ffb700 1 -- 172.21.15.103:0/4167450575 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f1fbc046970 con 0x7f1fc0132f50 2023-07-02T09:03:45.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.620+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc0061ea0 con 0x7f1fc0132f50 2023-07-02T09:03:45.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.620+0000 7f1fc4c91700 1 --2- 172.21.15.103:0/4167450575 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1fa40377c0 0x7f1fa4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:45.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.620+0000 7f1fc4c91700 1 --2- 172.21.15.103:0/4167450575 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1fa40377c0 0x7f1fa4039cd0 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:45.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.625+0000 7f1fb5ffb700 1 -- 172.21.15.103:0/4167450575 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f1fbc009c80 con 0x7f1fc0132f50 2023-07-02T09:03:45.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.818+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1fc00725b0 con 0x7f1fc0132f50 2023-07-02T09:03:45.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.818+0000 7f1fb5ffb700 1 -- 172.21.15.103:0/4167450575 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f1fbc02c020 con 0x7f1fc0132f50 2023-07-02T09:03:45.819 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1fa40377c0 msgr2=0x7f1fa4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 --2- 172.21.15.103:0/4167450575 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1fa40377c0 0x7f1fa4039cd0 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0132f50 msgr2=0x7f1fc01ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 --2- 172.21.15.103:0/4167450575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0132f50 0x7f1fc01ba9b0 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 shutdown_connections 2023-07-02T09:03:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 --2- 172.21.15.103:0/4167450575 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1fa40377c0 0x7f1fa4039cd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:45.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 --2- 172.21.15.103:0/4167450575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc0132f50 0x7f1fc01ba9b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 >> 172.21.15.103:0/4167450575 conn(0x7f1fc00a2080 msgr2=0x7f1fc00a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 shutdown_connections 2023-07-02T09:03:45.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:45.820+0000 7f1fc76f6700 1 -- 172.21.15.103:0/4167450575 wait complete. 2023-07-02T09:03:45.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:46.835 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.973+0000 7fe338fe0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.978+0000 7fe338fe0700 1 Processor -- start 2023-07-02T09:03:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.978+0000 7fe338fe0700 1 -- start start 2023-07-02T09:03:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.979+0000 7fe338fe0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe334133360 0x7fe33409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.979+0000 7fe338fe0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33409e260 con 0x7fe334132f50 2023-07-02T09:03:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.979+0000 7fe338fe0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3340a1640 con 0x7fe334133360 2023-07-02T09:03:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.979+0000 7fe331d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe334133360 0x7fe33409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.979+0000 7fe331d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe334133360 0x7fe33409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48424/0 (socket says 172.21.15.103:48424) 2023-07-02T09:03:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.979+0000 7fe33259c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe334132f50 0x7fe33409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:40992/0 (socket says 172.21.15.103:40992) 2023-07-02T09:03:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.979+0000 7fe331d9b700 1 -- 172.21.15.103:0/2114955470 learned_addr learned my addr 172.21.15.103:0/2114955470 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe331d9b700 1 -- 172.21.15.103:0/2114955470 >> v1:172.21.15.103:6789/0 conn(0x7fe334132f50 legacy=0x7fe33409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe331d9b700 1 -- 172.21.15.103:0/2114955470 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe334132a20 con 0x7fe334133360 2023-07-02T09:03:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe33159a700 1 -- 172.21.15.103:0/2114955470 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423531848 0 0) 0x7fe33409e260 con 0x7fe334132f50 2023-07-02T09:03:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe331d9b700 1 --2- 172.21.15.103:0/2114955470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334133360 0x7fe33409cc90 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ad6e870db3d098 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe33159a700 1 -- 172.21.15.103:0/2114955470 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe324009d90 con 0x7fe334133360 2023-07-02T09:03:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe33159a700 1 -- 172.21.15.103:0/2114955470 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe32400e230 con 0x7fe334133360 2023-07-02T09:03:46.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe33159a700 1 -- 172.21.15.103:0/2114955470 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe32400e4d0 con 0x7fe334133360 2023-07-02T09:03:46.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe338fe0700 1 -- 172.21.15.103:0/2114955470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334133360 msgr2=0x7fe33409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:46.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.980+0000 7fe338fe0700 1 --2- 172.21.15.103:0/2114955470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334133360 0x7fe33409cc90 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:46.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.981+0000 7fe338fe0700 1 -- 172.21.15.103:0/2114955470 shutdown_connections 2023-07-02T09:03:46.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.981+0000 7fe338fe0700 1 --2- 172.21.15.103:0/2114955470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334133360 0x7fe33409cc90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:46.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.981+0000 7fe338fe0700 1 -- 172.21.15.103:0/2114955470 >> 172.21.15.103:0/2114955470 conn(0x7fe3340a2080 msgr2=0x7fe3340a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.981+0000 7fe338fe0700 1 -- 172.21.15.103:0/2114955470 shutdown_connections 2023-07-02T09:03:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.981+0000 7fe338fe0700 1 -- 172.21.15.103:0/2114955470 wait complete. 2023-07-02T09:03:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.981+0000 7fe338fe0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.982+0000 7fe338fe0700 1 Processor -- start 2023-07-02T09:03:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.982+0000 7fe338fe0700 1 -- start start 2023-07-02T09:03:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe338fe0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334132f50 0x7fe3341ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe338fe0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33406d370 con 0x7fe334132f50 2023-07-02T09:03:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe33259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334132f50 0x7fe3341ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:46.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe33259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334132f50 0x7fe3341ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48432/0 (socket says 172.21.15.103:48432) 2023-07-02T09:03:46.985 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe33259c700 1 -- 172.21.15.103:0/2251122672 learned_addr learned my addr 172.21.15.103:0/2251122672 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:46.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe33259c700 1 -- 172.21.15.103:0/2251122672 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33406dde0 con 0x7fe334132f50 2023-07-02T09:03:46.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe33259c700 1 --2- 172.21.15.103:0/2251122672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334132f50 0x7fe3341ba9b0 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:46.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe322ffd700 1 -- 172.21.15.103:0/2251122672 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe328004b80 con 0x7fe334132f50 2023-07-02T09:03:46.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe322ffd700 1 -- 172.21.15.103:0/2251122672 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe328004ce0 con 0x7fe334132f50 2023-07-02T09:03:46.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3341ba4f0 con 0x7fe334132f50 2023-07-02T09:03:46.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe322ffd700 1 -- 172.21.15.103:0/2251122672 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe32800c4f0 con 0x7fe334132f50 2023-07-02T09:03:46.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.983+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe334071e40 con 0x7fe334132f50 2023-07-02T09:03:46.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.984+0000 7fe322ffd700 1 -- 172.21.15.103:0/2251122672 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fe32800c650 con 0x7fe334132f50 2023-07-02T09:03:46.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.984+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe334061ea0 con 0x7fe334132f50 2023-07-02T09:03:46.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.984+0000 7fe322ffd700 1 --2- 172.21.15.103:0/2251122672 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3140377c0 0x7fe314039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:46.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.984+0000 7fe322ffd700 1 -- 172.21.15.103:0/2251122672 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fe328046970 con 0x7fe334132f50 2023-07-02T09:03:46.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.985+0000 7fe331d9b700 1 --2- 172.21.15.103:0/2251122672 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3140377c0 0x7fe314039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:46.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.985+0000 7fe331d9b700 1 --2- 172.21.15.103:0/2251122672 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3140377c0 0x7fe314039cd0 crc :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:46.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:46.989+0000 7fe322ffd700 1 -- 172.21.15.103:0/2251122672 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fe328009c80 con 0x7fe334132f50 2023-07-02T09:03:47.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.175+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe3340725b0 con 0x7fe334132f50 2023-07-02T09:03:47.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.175+0000 7fe322ffd700 1 -- 172.21.15.103:0/2251122672 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fe3280098a0 con 0x7fe334132f50 2023-07-02T09:03:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:47.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.177+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3140377c0 msgr2=0x7fe314039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:47.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.177+0000 7fe338fe0700 1 --2- 172.21.15.103:0/2251122672 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3140377c0 0x7fe314039cd0 crc :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:47.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.177+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334132f50 msgr2=0x7fe3341ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:47.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.177+0000 7fe338fe0700 1 --2- 172.21.15.103:0/2251122672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334132f50 0x7fe3341ba9b0 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:47.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.178+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 shutdown_connections 2023-07-02T09:03:47.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.178+0000 7fe338fe0700 1 --2- 172.21.15.103:0/2251122672 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe3140377c0 0x7fe314039cd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:47.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.178+0000 7fe338fe0700 1 --2- 172.21.15.103:0/2251122672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe334132f50 0x7fe3341ba9b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:47.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.178+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 >> 172.21.15.103:0/2251122672 conn(0x7fe3340a2080 msgr2=0x7fe3340a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:47.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.178+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 shutdown_connections 2023-07-02T09:03:47.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:47.178+0000 7fe338fe0700 1 -- 172.21.15.103:0/2251122672 wait complete. 2023-07-02T09:03:47.190 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:48.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~242s 2023-07-02T09:03:48.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:48.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.327+0000 7f88473c0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:48.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.332+0000 7f88473c0700 1 Processor -- start 2023-07-02T09:03:48.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.332+0000 7f88473c0700 1 -- start start 2023-07-02T09:03:48.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f88473c0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8840132f50 0x7f8840133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:48.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f88473c0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884009e260 con 0x7f884009c580 2023-07-02T09:03:48.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f88473c0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88400a1640 con 0x7f8840132f50 2023-07-02T09:03:48.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f884515c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8840132f50 0x7f8840133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:48.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f884495b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f884009c580 0x7f884009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41004/0 (socket says 172.21.15.103:41004) 2023-07-02T09:03:48.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f884495b700 1 -- 172.21.15.103:0/2072491743 learned_addr learned my addr 172.21.15.103:0/2072491743 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:48.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f884515c700 1 -- 172.21.15.103:0/2072491743 >> v1:172.21.15.103:6789/0 conn(0x7f884009c580 legacy=0x7f884009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:48.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f8837fff700 1 -- 172.21.15.103:0/2072491743 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570587455 0 0) 0x7f884009e260 con 0x7f884009c580 2023-07-02T09:03:48.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.333+0000 7f884515c700 1 -- 172.21.15.103:0/2072491743 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8840132a20 con 0x7f8840132f50 2023-07-02T09:03:48.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f884515c700 1 --2- 172.21.15.103:0/2072491743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8840132f50 0x7f8840133360 crc :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569c3620f8dbe7fa server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:48.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f8837fff700 1 -- 172.21.15.103:0/2072491743 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8830009cd0 con 0x7f8840132f50 2023-07-02T09:03:48.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f8837fff700 1 -- 172.21.15.103:0/2072491743 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8830009e30 con 0x7f8840132f50 2023-07-02T09:03:48.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f8837fff700 1 -- 172.21.15.103:0/2072491743 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f883000e360 con 0x7f8840132f50 2023-07-02T09:03:48.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f88473c0700 1 -- 172.21.15.103:0/2072491743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8840132f50 msgr2=0x7f8840133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:48.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f88473c0700 1 --2- 172.21.15.103:0/2072491743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8840132f50 0x7f8840133360 crc :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:48.336 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f88473c0700 1 -- 172.21.15.103:0/2072491743 shutdown_connections 2023-07-02T09:03:48.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f88473c0700 1 --2- 172.21.15.103:0/2072491743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8840132f50 0x7f8840133360 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:48.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.334+0000 7f88473c0700 1 -- 172.21.15.103:0/2072491743 >> 172.21.15.103:0/2072491743 conn(0x7f88400a2080 msgr2=0x7f88400a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:48.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.335+0000 7f88473c0700 1 -- 172.21.15.103:0/2072491743 shutdown_connections 2023-07-02T09:03:48.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.335+0000 7f88473c0700 1 -- 172.21.15.103:0/2072491743 wait complete. 2023-07-02T09:03:48.337 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.335+0000 7f88473c0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.336+0000 7f88473c0700 1 Processor -- start 2023-07-02T09:03:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.336+0000 7f88473c0700 1 -- start start 2023-07-02T09:03:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.336+0000 7f88473c0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f884009c580 0x7f88401baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.336+0000 7f88473c0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88401ba610 con 0x7f884009c580 2023-07-02T09:03:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.336+0000 7f884515c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f884009c580 0x7f88401baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.336+0000 7f884515c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f884009c580 0x7f88401baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48448/0 (socket says 172.21.15.103:48448) 2023-07-02T09:03:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.336+0000 7f884515c700 1 -- 172.21.15.103:0/1465505261 learned_addr learned my addr 172.21.15.103:0/1465505261 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.337+0000 7f884515c700 1 -- 172.21.15.103:0/1465505261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884006dfc0 con 0x7f884009c580 2023-07-02T09:03:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.337+0000 7f884515c700 1 --2- 172.21.15.103:0/1465505261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f884009c580 0x7f88401baa60 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.337+0000 7f8835ffb700 1 -- 172.21.15.103:0/1465505261 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8830013530 con 0x7f884009c580 2023-07-02T09:03:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.337+0000 7f8835ffb700 1 -- 172.21.15.103:0/1465505261 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8830013690 con 0x7f884009c580 2023-07-02T09:03:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.337+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88401ba270 con 0x7f884009c580 2023-07-02T09:03:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.337+0000 7f8835ffb700 1 -- 172.21.15.103:0/1465505261 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8830013930 con 0x7f884009c580 2023-07-02T09:03:48.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.337+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f884006dba0 con 0x7f884009c580 2023-07-02T09:03:48.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.338+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88401ba460 con 0x7f884009c580 2023-07-02T09:03:48.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.338+0000 7f8835ffb700 1 -- 172.21.15.103:0/1465505261 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f8830013ae0 con 0x7f884009c580 2023-07-02T09:03:48.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.338+0000 7f8835ffb700 1 --2- 172.21.15.103:0/1465505261 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f88240377b0 0x7f8824039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:48.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.338+0000 7f8835ffb700 1 -- 172.21.15.103:0/1465505261 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f8830018040 con 0x7f884009c580 2023-07-02T09:03:48.341 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.339+0000 7f884495b700 1 --2- 172.21.15.103:0/1465505261 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f88240377b0 0x7f8824039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:48.342 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.339+0000 7f884495b700 1 --2- 172.21.15.103:0/1465505261 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f88240377b0 0x7f8824039cc0 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:48.343 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.343+0000 7f8835ffb700 1 -- 172.21.15.103:0/1465505261 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f8830018e00 con 0x7f884009c580 2023-07-02T09:03:48.530 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.530+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8840061ea0 con 0x7f884009c580 2023-07-02T09:03:48.530 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.530+0000 7f8835ffb700 1 -- 172.21.15.103:0/1465505261 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f8830010e60 con 0x7f884009c580 2023-07-02T09:03:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:48.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f88240377b0 msgr2=0x7f8824039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:48.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 --2- 172.21.15.103:0/1465505261 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f88240377b0 0x7f8824039cc0 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:48.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f884009c580 msgr2=0x7f88401baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:48.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 --2- 172.21.15.103:0/1465505261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f884009c580 0x7f88401baa60 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:48.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 shutdown_connections 2023-07-02T09:03:48.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 --2- 172.21.15.103:0/1465505261 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f88240377b0 0x7f8824039cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:48.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 --2- 172.21.15.103:0/1465505261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f884009c580 0x7f88401baa60 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:48.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 >> 172.21.15.103:0/1465505261 conn(0x7f88400a2080 msgr2=0x7f8840130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:48.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 shutdown_connections 2023-07-02T09:03:48.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:48.532+0000 7f88473c0700 1 -- 172.21.15.103:0/1465505261 wait complete. 2023-07-02T09:03:48.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:49.546 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:49.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.684+0000 7fe770370700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:49.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.689+0000 7fe770370700 1 Processor -- start 2023-07-02T09:03:49.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.689+0000 7fe770370700 1 -- start start 2023-07-02T09:03:49.690 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe770370700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe76812f250 0x7fe76812f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:49.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe770370700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76806df60 con 0x7fe768098a00 2023-07-02T09:03:49.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe770370700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76809a710 con 0x7fe76812f250 2023-07-02T09:03:49.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe76e10c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe76812f250 0x7fe76812f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:49.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe76d90b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe768098a00 0x7fe768098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41018/0 (socket says 172.21.15.103:41018) 2023-07-02T09:03:49.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe76d90b700 1 -- 172.21.15.103:0/606547203 learned_addr learned my addr 172.21.15.103:0/606547203 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:49.691 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe76e10c700 1 -- 172.21.15.103:0/606547203 >> v1:172.21.15.103:6789/0 conn(0x7fe768098a00 legacy=0x7fe768098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:49.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe76d10a700 1 -- 172.21.15.103:0/606547203 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924717330 0 0) 0x7fe76806df60 con 0x7fe768098a00 2023-07-02T09:03:49.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.690+0000 7fe76e10c700 1 -- 172.21.15.103:0/606547203 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76812ed20 con 0x7fe76812f250 2023-07-02T09:03:49.692 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.691+0000 7fe76e10c700 1 --2- 172.21.15.103:0/606547203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76812f250 0x7fe76812f660 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee6dc90319dfdd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:49.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.691+0000 7fe76d10a700 1 -- 172.21.15.103:0/606547203 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe758009cd0 con 0x7fe76812f250 2023-07-02T09:03:49.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.691+0000 7fe76d10a700 1 -- 172.21.15.103:0/606547203 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe758009e30 con 0x7fe76812f250 2023-07-02T09:03:49.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.691+0000 7fe76d10a700 1 -- 172.21.15.103:0/606547203 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe75800e360 con 0x7fe76812f250 2023-07-02T09:03:49.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.691+0000 7fe770370700 1 -- 172.21.15.103:0/606547203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76812f250 msgr2=0x7fe76812f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:49.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.691+0000 7fe770370700 1 --2- 172.21.15.103:0/606547203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76812f250 0x7fe76812f660 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:49.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.691+0000 7fe770370700 1 -- 172.21.15.103:0/606547203 shutdown_connections 2023-07-02T09:03:49.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.692+0000 7fe770370700 1 --2- 172.21.15.103:0/606547203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76812f250 0x7fe76812f660 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:49.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.692+0000 7fe770370700 1 -- 172.21.15.103:0/606547203 >> 172.21.15.103:0/606547203 conn(0x7fe76809d810 msgr2=0x7fe76809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:49.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.692+0000 7fe770370700 1 -- 172.21.15.103:0/606547203 shutdown_connections 2023-07-02T09:03:49.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.692+0000 7fe770370700 1 -- 172.21.15.103:0/606547203 wait complete. 2023-07-02T09:03:49.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.692+0000 7fe770370700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:49.695 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.693+0000 7fe770370700 1 Processor -- start 2023-07-02T09:03:49.695 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.693+0000 7fe770370700 1 -- start start 2023-07-02T09:03:49.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.693+0000 7fe770370700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe768098a00 0x7fe768133240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:49.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.693+0000 7fe770370700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681176a0 con 0x7fe768098a00 2023-07-02T09:03:49.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe76e10c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe768098a00 0x7fe768133240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:49.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe76e10c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe768098a00 0x7fe768133240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48480/0 (socket says 172.21.15.103:48480) 2023-07-02T09:03:49.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe76e10c700 1 -- 172.21.15.103:0/415018401 learned_addr learned my addr 172.21.15.103:0/415018401 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:49.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe76e10c700 1 -- 172.21.15.103:0/415018401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768117940 con 0x7fe768098a00 2023-07-02T09:03:49.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe76e10c700 1 --2- 172.21.15.103:0/415018401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe768098a00 0x7fe768133240 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:49.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe75effd700 1 -- 172.21.15.103:0/415018401 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe758013530 con 0x7fe768098a00 2023-07-02T09:03:49.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe75effd700 1 -- 172.21.15.103:0/415018401 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe758013690 con 0x7fe768098a00 2023-07-02T09:03:49.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe75effd700 1 -- 172.21.15.103:0/415018401 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe758013930 con 0x7fe768098a00 2023-07-02T09:03:49.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe768118690 con 0x7fe768098a00 2023-07-02T09:03:49.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.694+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7681173f0 con 0x7fe768098a00 2023-07-02T09:03:49.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.695+0000 7fe75effd700 1 -- 172.21.15.103:0/415018401 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fe758013ae0 con 0x7fe768098a00 2023-07-02T09:03:49.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.695+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe768134e50 con 0x7fe768098a00 2023-07-02T09:03:49.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.695+0000 7fe75effd700 1 --2- 172.21.15.103:0/415018401 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe74c0377b0 0x7fe74c039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:49.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.695+0000 7fe75effd700 1 -- 172.21.15.103:0/415018401 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fe758018040 con 0x7fe768098a00 2023-07-02T09:03:49.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.695+0000 7fe76d90b700 1 --2- 172.21.15.103:0/415018401 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe74c0377b0 0x7fe74c039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:49.699 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.696+0000 7fe76d90b700 1 --2- 172.21.15.103:0/415018401 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe74c0377b0 0x7fe74c039cc0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:49.700 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.700+0000 7fe75effd700 1 -- 172.21.15.103:0/415018401 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fe758018e00 con 0x7fe768098a00 2023-07-02T09:03:49.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.889+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe768061ea0 con 0x7fe768098a00 2023-07-02T09:03:49.890 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.890+0000 7fe75effd700 1 -- 172.21.15.103:0/415018401 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fe758010e60 con 0x7fe768098a00 2023-07-02T09:03:49.890 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:49.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe74c0377b0 msgr2=0x7fe74c039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:49.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 --2- 172.21.15.103:0/415018401 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe74c0377b0 0x7fe74c039cc0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:49.892 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe768098a00 msgr2=0x7fe768133240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 --2- 172.21.15.103:0/415018401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe768098a00 0x7fe768133240 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 shutdown_connections 2023-07-02T09:03:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 --2- 172.21.15.103:0/415018401 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe74c0377b0 0x7fe74c039cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 --2- 172.21.15.103:0/415018401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe768098a00 0x7fe768133240 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 >> 172.21.15.103:0/415018401 conn(0x7fe76809d810 msgr2=0x7fe76809f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 shutdown_connections 2023-07-02T09:03:49.893 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:49.892+0000 7fe770370700 1 -- 172.21.15.103:0/415018401 wait complete. 2023-07-02T09:03:49.905 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:50.905 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:51.047 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.043+0000 7fb897927700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:51.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.048+0000 7fb897927700 1 Processor -- start 2023-07-02T09:03:51.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.048+0000 7fb897927700 1 -- start start 2023-07-02T09:03:51.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb897927700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb890132030 0x7fb890134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:51.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb897927700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89006df60 con 0x7fb89012f250 2023-07-02T09:03:51.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb897927700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89009a710 con 0x7fb890132030 2023-07-02T09:03:51.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb8956c3700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb89012f250 0x7fb89012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41034/0 (socket says 172.21.15.103:41034) 2023-07-02T09:03:51.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb8956c3700 1 -- 172.21.15.103:0/540277074 learned_addr learned my addr 172.21.15.103:0/540277074 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:51.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb894ec2700 1 --2- 172.21.15.103:0/540277074 >> v2:172.21.15.103:3300/0 conn(0x7fb890132030 0x7fb890134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:51.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb894ec2700 1 -- 172.21.15.103:0/540277074 >> v1:172.21.15.103:6789/0 conn(0x7fb89012f250 legacy=0x7fb89012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:51.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb887fff700 1 -- 172.21.15.103:0/540277074 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898261465 0 0) 0x7fb89006df60 con 0x7fb89012f250 2023-07-02T09:03:51.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.049+0000 7fb894ec2700 1 -- 172.21.15.103:0/540277074 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89012ed20 con 0x7fb890132030 2023-07-02T09:03:51.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb894ec2700 1 --2- 172.21.15.103:0/540277074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb890132030 0x7fb890134450 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a654c4817876dd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:51.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb887fff700 1 -- 172.21.15.103:0/540277074 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb88c009cd0 con 0x7fb890132030 2023-07-02T09:03:51.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb887fff700 1 -- 172.21.15.103:0/540277074 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb88c009e30 con 0x7fb890132030 2023-07-02T09:03:51.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb887fff700 1 -- 172.21.15.103:0/540277074 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb88c00e360 con 0x7fb890132030 2023-07-02T09:03:51.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb897927700 1 -- 172.21.15.103:0/540277074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb890132030 msgr2=0x7fb890134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:51.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb897927700 1 --2- 172.21.15.103:0/540277074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb890132030 0x7fb890134450 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:51.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb897927700 1 -- 172.21.15.103:0/540277074 shutdown_connections 2023-07-02T09:03:51.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb897927700 1 --2- 172.21.15.103:0/540277074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb890132030 0x7fb890134450 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:51.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb897927700 1 -- 172.21.15.103:0/540277074 >> 172.21.15.103:0/540277074 conn(0x7fb89009d810 msgr2=0x7fb89009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:51.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.050+0000 7fb897927700 1 -- 172.21.15.103:0/540277074 shutdown_connections 2023-07-02T09:03:51.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.051+0000 7fb897927700 1 -- 172.21.15.103:0/540277074 wait complete. 2023-07-02T09:03:51.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.051+0000 7fb897927700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:51.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.052+0000 7fb897927700 1 Processor -- start 2023-07-02T09:03:51.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.052+0000 7fb897927700 1 -- start start 2023-07-02T09:03:51.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.052+0000 7fb897927700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89012f250 0x7fb890117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:51.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.052+0000 7fb897927700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb890118750 con 0x7fb89012f250 2023-07-02T09:03:51.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.052+0000 7fb8956c3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89012f250 0x7fb890117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:51.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.052+0000 7fb8956c3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89012f250 0x7fb890117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48504/0 (socket says 172.21.15.103:48504) 2023-07-02T09:03:51.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.052+0000 7fb8956c3700 1 -- 172.21.15.103:0/87462450 learned_addr learned my addr 172.21.15.103:0/87462450 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:51.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.053+0000 7fb8956c3700 1 -- 172.21.15.103:0/87462450 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89011a3c0 con 0x7fb89012f250 2023-07-02T09:03:51.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.053+0000 7fb8956c3700 1 --2- 172.21.15.103:0/87462450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89012f250 0x7fb890117360 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:51.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.053+0000 7fb885ffb700 1 -- 172.21.15.103:0/87462450 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb880004b80 con 0x7fb89012f250 2023-07-02T09:03:51.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.053+0000 7fb885ffb700 1 -- 172.21.15.103:0/87462450 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb880004ce0 con 0x7fb89012f250 2023-07-02T09:03:51.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.053+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb890119990 con 0x7fb89012f250 2023-07-02T09:03:51.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.053+0000 7fb885ffb700 1 -- 172.21.15.103:0/87462450 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb88000c4f0 con 0x7fb89012f250 2023-07-02T09:03:51.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.053+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb890119ce0 con 0x7fb89012f250 2023-07-02T09:03:51.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.054+0000 7fb885ffb700 1 -- 172.21.15.103:0/87462450 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb88000c650 con 0x7fb89012f250 2023-07-02T09:03:51.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.054+0000 7fb885ffb700 1 --2- 172.21.15.103:0/87462450 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb8740377c0 0x7fb874039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:51.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.054+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb890061ea0 con 0x7fb89012f250 2023-07-02T09:03:51.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.054+0000 7fb885ffb700 1 -- 172.21.15.103:0/87462450 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb8800469e0 con 0x7fb89012f250 2023-07-02T09:03:51.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.055+0000 7fb894ec2700 1 --2- 172.21.15.103:0/87462450 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb8740377c0 0x7fb874039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:51.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.055+0000 7fb894ec2700 1 --2- 172.21.15.103:0/87462450 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb8740377c0 0x7fb874039cd0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:51.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.059+0000 7fb885ffb700 1 -- 172.21.15.103:0/87462450 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb88000e1c0 con 0x7fb89012f250 2023-07-02T09:03:51.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.247+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb89012c6c0 con 0x7fb89012f250 2023-07-02T09:03:51.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.247+0000 7fb885ffb700 1 -- 172.21.15.103:0/87462450 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb8800098a0 con 0x7fb89012f250 2023-07-02T09:03:51.248 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:51.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb8740377c0 msgr2=0x7fb874039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:51.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 --2- 172.21.15.103:0/87462450 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb8740377c0 0x7fb874039cd0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:51.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89012f250 msgr2=0x7fb890117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:51.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 --2- 172.21.15.103:0/87462450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89012f250 0x7fb890117360 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:51.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 shutdown_connections 2023-07-02T09:03:51.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 --2- 172.21.15.103:0/87462450 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb8740377c0 0x7fb874039cd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 --2- 172.21.15.103:0/87462450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89012f250 0x7fb890117360 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 >> 172.21.15.103:0/87462450 conn(0x7fb89009d810 msgr2=0x7fb89012dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.249+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 shutdown_connections 2023-07-02T09:03:51.251 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:51.250+0000 7fb897927700 1 -- 172.21.15.103:0/87462450 wait complete. 2023-07-02T09:03:51.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:52.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:52.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.394+0000 7f2e1b77c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:52.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.399+0000 7f2e1b77c700 1 Processor -- start 2023-07-02T09:03:52.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.399+0000 7f2e1b77c700 1 -- start start 2023-07-02T09:03:52.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.399+0000 7f2e1b77c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2e1409a220 0x7f2e141348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:52.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.399+0000 7f2e1b77c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1409cea0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.399+0000 7f2e1b77c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1406d490 con 0x7f2e1409a220 2023-07-02T09:03:52.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e18d17700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2e1409a220 0x7f2e141348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:52.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e19518700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2e1406d6e0 0x7f2e14099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:41044/0 (socket says 172.21.15.103:41044) 2023-07-02T09:03:52.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e19518700 1 -- 172.21.15.103:0/4216417685 learned_addr learned my addr 172.21.15.103:0/4216417685 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:52.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e18d17700 1 -- 172.21.15.103:0/4216417685 >> v1:172.21.15.103:6789/0 conn(0x7f2e1406d6e0 legacy=0x7f2e14099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:52.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e0bfff700 1 -- 172.21.15.103:0/4216417685 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521736746 0 0) 0x7f2e1409cea0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e18d17700 1 -- 172.21.15.103:0/4216417685 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1406dc80 con 0x7f2e1409a220 2023-07-02T09:03:52.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e18d17700 1 --2- 172.21.15.103:0/4216417685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1409a220 0x7f2e141348e0 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa0d957e1ba3bc6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:52.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e0bfff700 1 -- 172.21.15.103:0/4216417685 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2e10009cd0 con 0x7f2e1409a220 2023-07-02T09:03:52.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e0bfff700 1 -- 172.21.15.103:0/4216417685 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2e10009e30 con 0x7f2e1409a220 2023-07-02T09:03:52.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.400+0000 7f2e0bfff700 1 -- 172.21.15.103:0/4216417685 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2e1000e360 con 0x7f2e1409a220 2023-07-02T09:03:52.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.401+0000 7f2e1b77c700 1 -- 172.21.15.103:0/4216417685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1409a220 msgr2=0x7f2e141348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:52.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.401+0000 7f2e1b77c700 1 --2- 172.21.15.103:0/4216417685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1409a220 0x7f2e141348e0 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:52.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.401+0000 7f2e1b77c700 1 -- 172.21.15.103:0/4216417685 shutdown_connections 2023-07-02T09:03:52.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.401+0000 7f2e1b77c700 1 --2- 172.21.15.103:0/4216417685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1409a220 0x7f2e141348e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:52.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.401+0000 7f2e1b77c700 1 -- 172.21.15.103:0/4216417685 >> 172.21.15.103:0/4216417685 conn(0x7f2e1409d810 msgr2=0x7f2e1409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:52.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.401+0000 7f2e1b77c700 1 -- 172.21.15.103:0/4216417685 shutdown_connections 2023-07-02T09:03:52.404 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.401+0000 7f2e1b77c700 1 -- 172.21.15.103:0/4216417685 wait complete. 2023-07-02T09:03:52.404 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.401+0000 7f2e1b77c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:52.404 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.402+0000 7f2e1b77c700 1 Processor -- start 2023-07-02T09:03:52.404 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.402+0000 7f2e1b77c700 1 -- start start 2023-07-02T09:03:52.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e1b77c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1406d6e0 0x7f2e1411db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:52.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e1b77c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1411f0e0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e19518700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1406d6e0 0x7f2e1411db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:52.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e19518700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1406d6e0 0x7f2e1411db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48514/0 (socket says 172.21.15.103:48514) 2023-07-02T09:03:52.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e19518700 1 -- 172.21.15.103:0/726748284 learned_addr learned my addr 172.21.15.103:0/726748284 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:52.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e19518700 1 -- 172.21.15.103:0/726748284 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e141244e0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e19518700 1 --2- 172.21.15.103:0/726748284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1406d6e0 0x7f2e1411db90 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:52.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e09ffb700 1 -- 172.21.15.103:0/726748284 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2e04004b80 con 0x7f2e1406d6e0 2023-07-02T09:03:52.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e09ffb700 1 -- 172.21.15.103:0/726748284 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2e04004ce0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e09ffb700 1 -- 172.21.15.103:0/726748284 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2e0400c550 con 0x7f2e1406d6e0 2023-07-02T09:03:52.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.403+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14124fa0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.404+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1412f2d0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.404+0000 7f2e09ffb700 1 -- 172.21.15.103:0/726748284 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2e0400c6b0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.404+0000 7f2e09ffb700 1 --2- 172.21.15.103:0/726748284 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2df80377c0 0x7f2df8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:52.407 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.404+0000 7f2e09ffb700 1 -- 172.21.15.103:0/726748284 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2e04046a50 con 0x7f2e1406d6e0 2023-07-02T09:03:52.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.404+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e14061ea0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.405+0000 7f2e18d17700 1 --2- 172.21.15.103:0/726748284 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2df80377c0 0x7f2df8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:52.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.405+0000 7f2e18d17700 1 --2- 172.21.15.103:0/726748284 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2df80377c0 0x7f2df8039cd0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:52.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.409+0000 7f2e09ffb700 1 -- 172.21.15.103:0/726748284 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2e04009c80 con 0x7f2e1406d6e0 2023-07-02T09:03:52.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.596+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2e14070fe0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.596+0000 7f2e09ffb700 1 -- 172.21.15.103:0/726748284 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2e040098a0 con 0x7f2e1406d6e0 2023-07-02T09:03:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:52.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2df80377c0 msgr2=0x7f2df8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:52.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 --2- 172.21.15.103:0/726748284 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2df80377c0 0x7f2df8039cd0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:52.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1406d6e0 msgr2=0x7f2e1411db90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:52.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 --2- 172.21.15.103:0/726748284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1406d6e0 0x7f2e1411db90 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:52.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 shutdown_connections 2023-07-02T09:03:52.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 --2- 172.21.15.103:0/726748284 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2df80377c0 0x7f2df8039cd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:52.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 --2- 172.21.15.103:0/726748284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e1406d6e0 0x7f2e1411db90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:52.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 >> 172.21.15.103:0/726748284 conn(0x7f2e1409d810 msgr2=0x7f2e14117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:52.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 shutdown_connections 2023-07-02T09:03:52.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:52.598+0000 7f2e1b77c700 1 -- 172.21.15.103:0/726748284 wait complete. 2023-07-02T09:03:52.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:53.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~248s 2023-07-02T09:03:53.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:53.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.750+0000 7f623861f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:53.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.755+0000 7f623861f700 1 Processor -- start 2023-07-02T09:03:53.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.755+0000 7f623861f700 1 -- start start 2023-07-02T09:03:53.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f623861f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6230132030 0x7f6230134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:53.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f623861f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f623006df60 con 0x7f623012f250 2023-07-02T09:03:53.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f623861f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623009a710 con 0x7f6230132030 2023-07-02T09:03:53.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f6235bba700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6230132030 0x7f6230134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:53.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f6235bba700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6230132030 0x7f6230134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58330/0 (socket says 172.21.15.103:58330) 2023-07-02T09:03:53.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f6235bba700 1 -- 172.21.15.103:0/3466622535 learned_addr learned my addr 172.21.15.103:0/3466622535 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:53.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f62363bb700 1 --1- 172.21.15.103:0/3466622535 >> v1:172.21.15.103:6789/0 conn(0x7f623012f250 0x7f623012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57236/0 (socket says 172.21.15.103:57236) 2023-07-02T09:03:53.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f6235bba700 1 -- 172.21.15.103:0/3466622535 >> v1:172.21.15.103:6789/0 conn(0x7f623012f250 legacy=0x7f623012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:53.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f6235bba700 1 -- 172.21.15.103:0/3466622535 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623012ed20 con 0x7f6230132030 2023-07-02T09:03:53.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f6235bba700 1 --2- 172.21.15.103:0/3466622535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6230132030 0x7f6230134450 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e637e46327d6bb server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:53.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f62353b9700 1 -- 172.21.15.103:0/3466622535 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f622c009d90 con 0x7f6230132030 2023-07-02T09:03:53.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f62353b9700 1 -- 172.21.15.103:0/3466622535 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f622c00e230 con 0x7f6230132030 2023-07-02T09:03:53.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.756+0000 7f62353b9700 1 -- 172.21.15.103:0/3466622535 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f622c00e4d0 con 0x7f6230132030 2023-07-02T09:03:53.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.757+0000 7f623861f700 1 -- 172.21.15.103:0/3466622535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6230132030 msgr2=0x7f6230134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:53.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.757+0000 7f623861f700 1 --2- 172.21.15.103:0/3466622535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6230132030 0x7f6230134450 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:53.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.757+0000 7f623861f700 1 -- 172.21.15.103:0/3466622535 shutdown_connections 2023-07-02T09:03:53.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.757+0000 7f623861f700 1 --2- 172.21.15.103:0/3466622535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6230132030 0x7f6230134450 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:53.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.757+0000 7f623861f700 1 -- 172.21.15.103:0/3466622535 >> 172.21.15.103:0/3466622535 conn(0x7f623009d810 msgr2=0x7f623009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:53.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.757+0000 7f623861f700 1 -- 172.21.15.103:0/3466622535 shutdown_connections 2023-07-02T09:03:53.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.757+0000 7f623861f700 1 -- 172.21.15.103:0/3466622535 wait complete. 2023-07-02T09:03:53.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.757+0000 7f623861f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:53.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.758+0000 7f623861f700 1 Processor -- start 2023-07-02T09:03:53.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.758+0000 7f623861f700 1 -- start start 2023-07-02T09:03:53.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.759+0000 7f623861f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f623012f250 0x7f6230071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:53.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.759+0000 7f623861f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6230117640 con 0x7f623012f250 2023-07-02T09:03:53.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.759+0000 7f62363bb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f623012f250 0x7f6230071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:53.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.759+0000 7f62363bb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f623012f250 0x7f6230071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58338/0 (socket says 172.21.15.103:58338) 2023-07-02T09:03:53.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.759+0000 7f62363bb700 1 -- 172.21.15.103:0/3049370925 learned_addr learned my addr 172.21.15.103:0/3049370925 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:53.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.759+0000 7f62363bb700 1 -- 172.21.15.103:0/3049370925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6230130050 con 0x7f623012f250 2023-07-02T09:03:53.762 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.760+0000 7f62363bb700 1 --2- 172.21.15.103:0/3049370925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f623012f250 0x7f6230071030 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:53.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.760+0000 7f6226ffd700 1 -- 172.21.15.103:0/3049370925 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f62200059a0 con 0x7f623012f250 2023-07-02T09:03:53.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.760+0000 7f6226ffd700 1 -- 172.21.15.103:0/3049370925 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6220005b00 con 0x7f623012f250 2023-07-02T09:03:53.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.760+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6230130970 con 0x7f623012f250 2023-07-02T09:03:53.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.760+0000 7f6226ffd700 1 -- 172.21.15.103:0/3049370925 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6220005da0 con 0x7f623012f250 2023-07-02T09:03:53.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.760+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6230119130 con 0x7f623012f250 2023-07-02T09:03:53.763 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.761+0000 7f6226ffd700 1 -- 172.21.15.103:0/3049370925 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f62200070b0 con 0x7f623012f250 2023-07-02T09:03:53.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.761+0000 7f6226ffd700 1 --2- 172.21.15.103:0/3049370925 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f62140377c0 0x7f6214039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:53.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.761+0000 7f6226ffd700 1 -- 172.21.15.103:0/3049370925 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6220046ae0 con 0x7f623012f250 2023-07-02T09:03:53.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.761+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6230117e00 con 0x7f623012f250 2023-07-02T09:03:53.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.761+0000 7f6235bba700 1 --2- 172.21.15.103:0/3049370925 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f62140377c0 0x7f6214039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:53.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.762+0000 7f6235bba700 1 --2- 172.21.15.103:0/3049370925 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f62140377c0 0x7f6214039cd0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:53.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.766+0000 7f6226ffd700 1 -- 172.21.15.103:0/3049370925 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f622000a6a0 con 0x7f623012f250 2023-07-02T09:03:53.953 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.953+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6230061ea0 con 0x7f623012f250 2023-07-02T09:03:53.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.954+0000 7f6226ffd700 1 -- 172.21.15.103:0/3049370925 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f622000a4a0 con 0x7f623012f250 2023-07-02T09:03:53.954 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:53.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f62140377c0 msgr2=0x7f6214039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:53.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 --2- 172.21.15.103:0/3049370925 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f62140377c0 0x7f6214039cd0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:53.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f623012f250 msgr2=0x7f6230071030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:53.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 --2- 172.21.15.103:0/3049370925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f623012f250 0x7f6230071030 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:53.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 shutdown_connections 2023-07-02T09:03:53.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 --2- 172.21.15.103:0/3049370925 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f62140377c0 0x7f6214039cd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:53.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 --2- 172.21.15.103:0/3049370925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f623012f250 0x7f6230071030 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:53.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 >> 172.21.15.103:0/3049370925 conn(0x7f623009d810 msgr2=0x7f6230128a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:53.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 shutdown_connections 2023-07-02T09:03:53.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:53.956+0000 7f623861f700 1 -- 172.21.15.103:0/3049370925 wait complete. 2023-07-02T09:03:53.969 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:54.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:55.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.107+0000 7f795dacc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:55.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.113+0000 7f795dacc700 1 Processor -- start 2023-07-02T09:03:55.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.113+0000 7f795dacc700 1 -- start start 2023-07-02T09:03:55.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.113+0000 7f795dacc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7958132f50 0x7f7958133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:55.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.113+0000 7f795dacc700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795809e260 con 0x7f795809c580 2023-07-02T09:03:55.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.113+0000 7f795dacc700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79580a1640 con 0x7f7958132f50 2023-07-02T09:03:55.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.113+0000 7f79577fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7958132f50 0x7f7958133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:55.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f7956ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f795809c580 0x7f795809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57244/0 (socket says 172.21.15.103:57244) 2023-07-02T09:03:55.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f79577fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7958132f50 0x7f7958133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58348/0 (socket says 172.21.15.103:58348) 2023-07-02T09:03:55.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f7956ffd700 1 -- 172.21.15.103:0/4255578333 learned_addr learned my addr 172.21.15.103:0/4255578333 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:55.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f79577fe700 1 -- 172.21.15.103:0/4255578333 >> v1:172.21.15.103:6789/0 conn(0x7f795809c580 legacy=0x7f795809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:55.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f79577fe700 1 -- 172.21.15.103:0/4255578333 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958132a20 con 0x7f7958132f50 2023-07-02T09:03:55.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f79577fe700 1 --2- 172.21.15.103:0/4255578333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7958132f50 0x7f7958133360 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5391e6cb514835a5 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:55.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f79567fc700 1 -- 172.21.15.103:0/4255578333 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f794c009cd0 con 0x7f7958132f50 2023-07-02T09:03:55.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f79567fc700 1 -- 172.21.15.103:0/4255578333 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f794c009e30 con 0x7f7958132f50 2023-07-02T09:03:55.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.114+0000 7f79567fc700 1 -- 172.21.15.103:0/4255578333 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f794c00e360 con 0x7f7958132f50 2023-07-02T09:03:55.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.115+0000 7f795dacc700 1 -- 172.21.15.103:0/4255578333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7958132f50 msgr2=0x7f7958133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:55.117 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.115+0000 7f795dacc700 1 --2- 172.21.15.103:0/4255578333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7958132f50 0x7f7958133360 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:55.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.115+0000 7f795dacc700 1 -- 172.21.15.103:0/4255578333 shutdown_connections 2023-07-02T09:03:55.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.115+0000 7f795dacc700 1 --2- 172.21.15.103:0/4255578333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7958132f50 0x7f7958133360 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:55.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.115+0000 7f795dacc700 1 -- 172.21.15.103:0/4255578333 >> 172.21.15.103:0/4255578333 conn(0x7f79580a2080 msgr2=0x7f79580a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:55.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.115+0000 7f795dacc700 1 -- 172.21.15.103:0/4255578333 shutdown_connections 2023-07-02T09:03:55.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.115+0000 7f795dacc700 1 -- 172.21.15.103:0/4255578333 wait complete. 2023-07-02T09:03:55.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.115+0000 7f795dacc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:55.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.116+0000 7f795dacc700 1 Processor -- start 2023-07-02T09:03:55.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.116+0000 7f795dacc700 1 -- start start 2023-07-02T09:03:55.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.117+0000 7f795dacc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f795809c580 0x7f795812b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:55.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.117+0000 7f795dacc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795806f9d0 con 0x7f795809c580 2023-07-02T09:03:55.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.117+0000 7f79577fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f795809c580 0x7f795812b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:55.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.117+0000 7f79577fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f795809c580 0x7f795812b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58354/0 (socket says 172.21.15.103:58354) 2023-07-02T09:03:55.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.117+0000 7f79577fe700 1 -- 172.21.15.103:0/2503646906 learned_addr learned my addr 172.21.15.103:0/2503646906 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:55.119 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.117+0000 7f79577fe700 1 -- 172.21.15.103:0/2503646906 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958133a00 con 0x7f795809c580 2023-07-02T09:03:55.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.117+0000 7f79577fe700 1 --2- 172.21.15.103:0/2503646906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f795809c580 0x7f795812b7b0 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:55.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.118+0000 7f793ffff700 1 -- 172.21.15.103:0/2503646906 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f794c013530 con 0x7f795809c580 2023-07-02T09:03:55.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.118+0000 7f793ffff700 1 -- 172.21.15.103:0/2503646906 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f794c013690 con 0x7f795809c580 2023-07-02T09:03:55.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.118+0000 7f793ffff700 1 -- 172.21.15.103:0/2503646906 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f794c013930 con 0x7f795809c580 2023-07-02T09:03:55.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.118+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795806d160 con 0x7f795809c580 2023-07-02T09:03:55.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.118+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795806da70 con 0x7f795809c580 2023-07-02T09:03:55.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.118+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958061ea0 con 0x7f795809c580 2023-07-02T09:03:55.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.119+0000 7f793ffff700 1 -- 172.21.15.103:0/2503646906 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f794c013ae0 con 0x7f795809c580 2023-07-02T09:03:55.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.119+0000 7f793ffff700 1 --2- 172.21.15.103:0/2503646906 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f79440377c0 0x7f7944039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:55.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.119+0000 7f793ffff700 1 -- 172.21.15.103:0/2503646906 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f794c018040 con 0x7f795809c580 2023-07-02T09:03:55.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.119+0000 7f7956ffd700 1 --2- 172.21.15.103:0/2503646906 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f79440377c0 0x7f7944039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:55.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.120+0000 7f7956ffd700 1 --2- 172.21.15.103:0/2503646906 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f79440377c0 0x7f7944039cd0 crc :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:55.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.124+0000 7f793ffff700 1 -- 172.21.15.103:0/2503646906 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f794c018200 con 0x7f795809c580 2023-07-02T09:03:55.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.311+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f79580725b0 con 0x7f795809c580 2023-07-02T09:03:55.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.311+0000 7f793ffff700 1 -- 172.21.15.103:0/2503646906 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f794c00f2a0 con 0x7f795809c580 2023-07-02T09:03:55.311 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:55.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f79440377c0 msgr2=0x7f7944039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:55.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 --2- 172.21.15.103:0/2503646906 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f79440377c0 0x7f7944039cd0 crc :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:55.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f795809c580 msgr2=0x7f795812b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:55.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 --2- 172.21.15.103:0/2503646906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f795809c580 0x7f795812b7b0 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:55.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 shutdown_connections 2023-07-02T09:03:55.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 --2- 172.21.15.103:0/2503646906 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f79440377c0 0x7f7944039cd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:55.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 --2- 172.21.15.103:0/2503646906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f795809c580 0x7f795812b7b0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:55.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 >> 172.21.15.103:0/2503646906 conn(0x7f79580a2080 msgr2=0x7f79581babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:55.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 shutdown_connections 2023-07-02T09:03:55.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:55.313+0000 7f795dacc700 1 -- 172.21.15.103:0/2503646906 wait complete. 2023-07-02T09:03:55.326 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:56.327 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:56.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.466+0000 7ff592c2e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:56.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.471+0000 7ff592c2e700 1 Processor -- start 2023-07-02T09:03:56.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.471+0000 7ff592c2e700 1 -- start start 2023-07-02T09:03:56.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff592c2e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff58c133360 0x7ff58c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:56.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff592c2e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c09e260 con 0x7ff58c132f50 2023-07-02T09:03:56.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff592c2e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c0a1640 con 0x7ff58c133360 2023-07-02T09:03:56.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff58bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff58c133360 0x7ff58c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:56.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff5909ca700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff58c132f50 0x7ff58c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57258/0 (socket says 172.21.15.103:57258) 2023-07-02T09:03:56.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff58bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff58c133360 0x7ff58c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58356/0 (socket says 172.21.15.103:58356) 2023-07-02T09:03:56.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff5909ca700 1 -- 172.21.15.103:0/3954252366 learned_addr learned my addr 172.21.15.103:0/3954252366 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:56.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff58bfff700 1 -- 172.21.15.103:0/3954252366 >> v1:172.21.15.103:6789/0 conn(0x7ff58c132f50 legacy=0x7ff58c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:56.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff58bfff700 1 -- 172.21.15.103:0/3954252366 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c132a20 con 0x7ff58c133360 2023-07-02T09:03:56.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.472+0000 7ff58b7fe700 1 -- 172.21.15.103:0/3954252366 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929775654 0 0) 0x7ff58c09e260 con 0x7ff58c132f50 2023-07-02T09:03:56.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff58bfff700 1 --2- 172.21.15.103:0/3954252366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c133360 0x7ff58c09cc90 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3faff690f40a3d8a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:56.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff58b7fe700 1 -- 172.21.15.103:0/3954252366 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff578009cd0 con 0x7ff58c133360 2023-07-02T09:03:56.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff58b7fe700 1 -- 172.21.15.103:0/3954252366 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff578009e30 con 0x7ff58c133360 2023-07-02T09:03:56.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff58b7fe700 1 -- 172.21.15.103:0/3954252366 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff57800e360 con 0x7ff58c133360 2023-07-02T09:03:56.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff592c2e700 1 -- 172.21.15.103:0/3954252366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c133360 msgr2=0x7ff58c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:56.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff592c2e700 1 --2- 172.21.15.103:0/3954252366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c133360 0x7ff58c09cc90 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:56.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff592c2e700 1 -- 172.21.15.103:0/3954252366 shutdown_connections 2023-07-02T09:03:56.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff592c2e700 1 --2- 172.21.15.103:0/3954252366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c133360 0x7ff58c09cc90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:56.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff592c2e700 1 -- 172.21.15.103:0/3954252366 >> 172.21.15.103:0/3954252366 conn(0x7ff58c0a2080 msgr2=0x7ff58c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:56.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.473+0000 7ff592c2e700 1 -- 172.21.15.103:0/3954252366 shutdown_connections 2023-07-02T09:03:56.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.474+0000 7ff592c2e700 1 -- 172.21.15.103:0/3954252366 wait complete. 2023-07-02T09:03:56.477 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.474+0000 7ff592c2e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:56.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.475+0000 7ff592c2e700 1 Processor -- start 2023-07-02T09:03:56.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.475+0000 7ff592c2e700 1 -- start start 2023-07-02T09:03:56.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.475+0000 7ff592c2e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c132f50 0x7ff58c12c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:56.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.475+0000 7ff592c2e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c06d2f0 con 0x7ff58c132f50 2023-07-02T09:03:56.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff5909ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c132f50 0x7ff58c12c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:56.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff5909ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c132f50 0x7ff58c12c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58364/0 (socket says 172.21.15.103:58364) 2023-07-02T09:03:56.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff5909ca700 1 -- 172.21.15.103:0/3737145346 learned_addr learned my addr 172.21.15.103:0/3737145346 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:56.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff5909ca700 1 -- 172.21.15.103:0/3737145346 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c06dd60 con 0x7ff58c132f50 2023-07-02T09:03:56.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff5909ca700 1 --2- 172.21.15.103:0/3737145346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c132f50 0x7ff58c12c1a0 crc :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:56.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff5897fa700 1 -- 172.21.15.103:0/3737145346 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff580004b80 con 0x7ff58c132f50 2023-07-02T09:03:56.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff5897fa700 1 -- 172.21.15.103:0/3737145346 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff580004ce0 con 0x7ff58c132f50 2023-07-02T09:03:56.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58c12bc50 con 0x7ff58c132f50 2023-07-02T09:03:56.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff5897fa700 1 -- 172.21.15.103:0/3737145346 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff58000c4f0 con 0x7ff58c132f50 2023-07-02T09:03:56.479 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.476+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c06f850 con 0x7ff58c132f50 2023-07-02T09:03:56.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.477+0000 7ff5897fa700 1 -- 172.21.15.103:0/3737145346 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7ff58000c650 con 0x7ff58c132f50 2023-07-02T09:03:56.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.477+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c06dbb0 con 0x7ff58c132f50 2023-07-02T09:03:56.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.477+0000 7ff5897fa700 1 --2- 172.21.15.103:0/3737145346 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff5700377c0 0x7ff570039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:56.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.477+0000 7ff5897fa700 1 -- 172.21.15.103:0/3737145346 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7ff580046970 con 0x7ff58c132f50 2023-07-02T09:03:56.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.478+0000 7ff58bfff700 1 --2- 172.21.15.103:0/3737145346 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff5700377c0 0x7ff570039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:56.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.478+0000 7ff58bfff700 1 --2- 172.21.15.103:0/3737145346 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff5700377c0 0x7ff570039cd0 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:56.482 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.482+0000 7ff5897fa700 1 -- 172.21.15.103:0/3737145346 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7ff580009c80 con 0x7ff58c132f50 2023-07-02T09:03:56.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.668+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff58c061ea0 con 0x7ff58c132f50 2023-07-02T09:03:56.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.669+0000 7ff5897fa700 1 -- 172.21.15.103:0/3737145346 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7ff5800098a0 con 0x7ff58c132f50 2023-07-02T09:03:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:56.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff5700377c0 msgr2=0x7ff570039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:56.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 --2- 172.21.15.103:0/3737145346 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff5700377c0 0x7ff570039cd0 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:56.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c132f50 msgr2=0x7ff58c12c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:56.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 --2- 172.21.15.103:0/3737145346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c132f50 0x7ff58c12c1a0 crc :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:56.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 shutdown_connections 2023-07-02T09:03:56.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 --2- 172.21.15.103:0/3737145346 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7ff5700377c0 0x7ff570039cd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:56.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 --2- 172.21.15.103:0/3737145346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff58c132f50 0x7ff58c12c1a0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:56.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 >> 172.21.15.103:0/3737145346 conn(0x7ff58c0a2080 msgr2=0x7ff58c130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:56.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 shutdown_connections 2023-07-02T09:03:56.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:56.671+0000 7ff592c2e700 1 -- 172.21.15.103:0/3737145346 wait complete. 2023-07-02T09:03:56.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:57.684 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:57.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.824+0000 7facb27e5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:57.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.829+0000 7facb27e5700 1 Processor -- start 2023-07-02T09:03:57.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.829+0000 7facb27e5700 1 -- start start 2023-07-02T09:03:57.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facb27e5700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7facac132f50 0x7facac133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:57.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facb27e5700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac09e260 con 0x7facac09c580 2023-07-02T09:03:57.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facb27e5700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facac0a1640 con 0x7facac132f50 2023-07-02T09:03:57.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facabfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7facac132f50 0x7facac133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:57.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facab7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7facac09c580 0x7facac09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57266/0 (socket says 172.21.15.103:57266) 2023-07-02T09:03:57.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facab7fe700 1 -- 172.21.15.103:0/1613133645 learned_addr learned my addr 172.21.15.103:0/1613133645 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:57.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facabfff700 1 -- 172.21.15.103:0/1613133645 >> v1:172.21.15.103:6789/0 conn(0x7facac09c580 legacy=0x7facac09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:57.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facabfff700 1 -- 172.21.15.103:0/1613133645 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac132a20 con 0x7facac132f50 2023-07-02T09:03:57.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.830+0000 7facabfff700 1 --2- 172.21.15.103:0/1613133645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac132f50 0x7facac133360 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404b2f410eeb3456 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:57.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facaaffd700 1 -- 172.21.15.103:0/1613133645 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fac9c009cd0 con 0x7facac132f50 2023-07-02T09:03:57.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facaaffd700 1 -- 172.21.15.103:0/1613133645 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fac9c009e30 con 0x7facac132f50 2023-07-02T09:03:57.832 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facaaffd700 1 -- 172.21.15.103:0/1613133645 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fac9c00e360 con 0x7facac132f50 2023-07-02T09:03:57.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facb27e5700 1 -- 172.21.15.103:0/1613133645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac132f50 msgr2=0x7facac133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:57.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facb27e5700 1 --2- 172.21.15.103:0/1613133645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac132f50 0x7facac133360 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:57.833 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facb27e5700 1 -- 172.21.15.103:0/1613133645 shutdown_connections 2023-07-02T09:03:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facb27e5700 1 --2- 172.21.15.103:0/1613133645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac132f50 0x7facac133360 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facb27e5700 1 -- 172.21.15.103:0/1613133645 >> 172.21.15.103:0/1613133645 conn(0x7facac0a2080 msgr2=0x7facac0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facb27e5700 1 -- 172.21.15.103:0/1613133645 shutdown_connections 2023-07-02T09:03:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.831+0000 7facb27e5700 1 -- 172.21.15.103:0/1613133645 wait complete. 2023-07-02T09:03:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.832+0000 7facb27e5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.833+0000 7facb27e5700 1 Processor -- start 2023-07-02T09:03:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.833+0000 7facb27e5700 1 -- start start 2023-07-02T09:03:57.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.833+0000 7facb27e5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac09c580 0x7facac12b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:57.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.833+0000 7facb27e5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac06f9d0 con 0x7facac09c580 2023-07-02T09:03:57.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7facabfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac09c580 0x7facac12b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:57.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7facabfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac09c580 0x7facac12b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58390/0 (socket says 172.21.15.103:58390) 2023-07-02T09:03:57.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7facabfff700 1 -- 172.21.15.103:0/2137602742 learned_addr learned my addr 172.21.15.103:0/2137602742 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:57.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7facabfff700 1 -- 172.21.15.103:0/2137602742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac133a00 con 0x7facac09c580 2023-07-02T09:03:57.836 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7facabfff700 1 --2- 172.21.15.103:0/2137602742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac09c580 0x7facac12b7b0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:57.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7faca8ff9700 1 -- 172.21.15.103:0/2137602742 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fac9c013530 con 0x7facac09c580 2023-07-02T09:03:57.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7faca8ff9700 1 -- 172.21.15.103:0/2137602742 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fac9c013690 con 0x7facac09c580 2023-07-02T09:03:57.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7faca8ff9700 1 -- 172.21.15.103:0/2137602742 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fac9c013930 con 0x7facac09c580 2023-07-02T09:03:57.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac06d160 con 0x7facac09c580 2023-07-02T09:03:57.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.834+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac06da70 con 0x7facac09c580 2023-07-02T09:03:57.837 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.835+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac061ea0 con 0x7facac09c580 2023-07-02T09:03:57.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.836+0000 7faca8ff9700 1 -- 172.21.15.103:0/2137602742 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fac9c013ae0 con 0x7facac09c580 2023-07-02T09:03:57.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.836+0000 7faca8ff9700 1 --2- 172.21.15.103:0/2137602742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac980377c0 0x7fac98039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:57.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.836+0000 7faca8ff9700 1 -- 172.21.15.103:0/2137602742 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fac9c018040 con 0x7facac09c580 2023-07-02T09:03:57.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.836+0000 7facab7fe700 1 --2- 172.21.15.103:0/2137602742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac980377c0 0x7fac98039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:57.838 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.837+0000 7facab7fe700 1 --2- 172.21.15.103:0/2137602742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac980377c0 0x7fac98039cd0 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:57.840 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:57.841+0000 7faca8ff9700 1 -- 172.21.15.103:0/2137602742 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fac9c018200 con 0x7facac09c580 2023-07-02T09:03:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.030+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7facac0725b0 con 0x7facac09c580 2023-07-02T09:03:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.030+0000 7faca8ff9700 1 -- 172.21.15.103:0/2137602742 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fac9c00f2a0 con 0x7facac09c580 2023-07-02T09:03:58.031 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:58.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.032+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac980377c0 msgr2=0x7fac98039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:58.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.032+0000 7facb27e5700 1 --2- 172.21.15.103:0/2137602742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac980377c0 0x7fac98039cd0 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:58.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.032+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac09c580 msgr2=0x7facac12b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:58.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.033+0000 7facb27e5700 1 --2- 172.21.15.103:0/2137602742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac09c580 0x7facac12b7b0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:58.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.033+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 shutdown_connections 2023-07-02T09:03:58.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.033+0000 7facb27e5700 1 --2- 172.21.15.103:0/2137602742 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac980377c0 0x7fac98039cd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:58.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.033+0000 7facb27e5700 1 --2- 172.21.15.103:0/2137602742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facac09c580 0x7facac12b7b0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:58.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.033+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 >> 172.21.15.103:0/2137602742 conn(0x7facac0a2080 msgr2=0x7facac1babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:58.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.033+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 shutdown_connections 2023-07-02T09:03:58.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:58.033+0000 7facb27e5700 1 -- 172.21.15.103:0/2137602742 wait complete. 2023-07-02T09:03:58.045 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:03:59.046 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:03:59.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~253s 2023-07-02T09:03:59.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.184+0000 7fe80786b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:59.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.189+0000 7fe80786b700 1 Processor -- start 2023-07-02T09:03:59.189 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.189+0000 7fe80786b700 1 -- start start 2023-07-02T09:03:59.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe80786b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe800133360 0x7fe80009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:59.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe80786b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80009e260 con 0x7fe800132f50 2023-07-02T09:03:59.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe80786b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8000a1640 con 0x7fe800133360 2023-07-02T09:03:59.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe804e06700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe800133360 0x7fe80009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:59.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe805607700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe800132f50 0x7fe80009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57270/0 (socket says 172.21.15.103:57270) 2023-07-02T09:03:59.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe804e06700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe800133360 0x7fe80009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58404/0 (socket says 172.21.15.103:58404) 2023-07-02T09:03:59.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe805607700 1 -- 172.21.15.103:0/3223857224 learned_addr learned my addr 172.21.15.103:0/3223857224 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:03:59.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe804e06700 1 -- 172.21.15.103:0/3223857224 >> v1:172.21.15.103:6789/0 conn(0x7fe800132f50 legacy=0x7fe80009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:59.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe804e06700 1 -- 172.21.15.103:0/3223857224 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe800132a20 con 0x7fe800133360 2023-07-02T09:03:59.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.190+0000 7fe7f7fff700 1 -- 172.21.15.103:0/3223857224 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893443792 0 0) 0x7fe80009e260 con 0x7fe800132f50 2023-07-02T09:03:59.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe804e06700 1 --2- 172.21.15.103:0/3223857224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800133360 0x7fe80009cc90 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ad0c44094ea837 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:59.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe7f7fff700 1 -- 172.21.15.103:0/3223857224 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe7fc009cd0 con 0x7fe800133360 2023-07-02T09:03:59.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe7f7fff700 1 -- 172.21.15.103:0/3223857224 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe7fc009e30 con 0x7fe800133360 2023-07-02T09:03:59.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe7f7fff700 1 -- 172.21.15.103:0/3223857224 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe7fc00e360 con 0x7fe800133360 2023-07-02T09:03:59.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe80786b700 1 -- 172.21.15.103:0/3223857224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800133360 msgr2=0x7fe80009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:59.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe80786b700 1 --2- 172.21.15.103:0/3223857224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800133360 0x7fe80009cc90 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:59.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe80786b700 1 -- 172.21.15.103:0/3223857224 shutdown_connections 2023-07-02T09:03:59.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe80786b700 1 --2- 172.21.15.103:0/3223857224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800133360 0x7fe80009cc90 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:59.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.191+0000 7fe80786b700 1 -- 172.21.15.103:0/3223857224 >> 172.21.15.103:0/3223857224 conn(0x7fe8000a2080 msgr2=0x7fe8000a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:59.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.192+0000 7fe80786b700 1 -- 172.21.15.103:0/3223857224 shutdown_connections 2023-07-02T09:03:59.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.192+0000 7fe80786b700 1 -- 172.21.15.103:0/3223857224 wait complete. 2023-07-02T09:03:59.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.192+0000 7fe80786b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:03:59.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.193+0000 7fe80786b700 1 Processor -- start 2023-07-02T09:03:59.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.193+0000 7fe80786b700 1 -- start start 2023-07-02T09:03:59.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.193+0000 7fe80786b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800132f50 0x7fe8001ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:59.196 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.193+0000 7fe80786b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80006d370 con 0x7fe800132f50 2023-07-02T09:03:59.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.193+0000 7fe805607700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800132f50 0x7fe8001ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:59.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.193+0000 7fe805607700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800132f50 0x7fe8001ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58408/0 (socket says 172.21.15.103:58408) 2023-07-02T09:03:59.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.193+0000 7fe805607700 1 -- 172.21.15.103:0/776573578 learned_addr learned my addr 172.21.15.103:0/776573578 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:03:59.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.194+0000 7fe805607700 1 -- 172.21.15.103:0/776573578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80006dde0 con 0x7fe800132f50 2023-07-02T09:03:59.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.194+0000 7fe805607700 1 --2- 172.21.15.103:0/776573578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800132f50 0x7fe8001ba9b0 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:59.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.194+0000 7fe7f5ffb700 1 -- 172.21.15.103:0/776573578 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe7f0004b80 con 0x7fe800132f50 2023-07-02T09:03:59.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.194+0000 7fe7f5ffb700 1 -- 172.21.15.103:0/776573578 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe7f0004ce0 con 0x7fe800132f50 2023-07-02T09:03:59.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.194+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8001ba4f0 con 0x7fe800132f50 2023-07-02T09:03:59.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.194+0000 7fe7f5ffb700 1 -- 172.21.15.103:0/776573578 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe7f000c4f0 con 0x7fe800132f50 2023-07-02T09:03:59.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.194+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe800071e40 con 0x7fe800132f50 2023-07-02T09:03:59.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.195+0000 7fe7f5ffb700 1 -- 172.21.15.103:0/776573578 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fe7f000c650 con 0x7fe800132f50 2023-07-02T09:03:59.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.195+0000 7fe7f5ffb700 1 --2- 172.21.15.103:0/776573578 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe7e40377c0 0x7fe7e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:03:59.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.195+0000 7fe7f5ffb700 1 -- 172.21.15.103:0/776573578 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fe7f0046970 con 0x7fe800132f50 2023-07-02T09:03:59.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.195+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe800061ea0 con 0x7fe800132f50 2023-07-02T09:03:59.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.195+0000 7fe804e06700 1 --2- 172.21.15.103:0/776573578 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe7e40377c0 0x7fe7e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:03:59.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.196+0000 7fe804e06700 1 --2- 172.21.15.103:0/776573578 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe7e40377c0 0x7fe7e4039cd0 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:03:59.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.200+0000 7fe7f5ffb700 1 -- 172.21.15.103:0/776573578 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fe7f0009c80 con 0x7fe800132f50 2023-07-02T09:03:59.387 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.387+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe8000725b0 con 0x7fe800132f50 2023-07-02T09:03:59.388 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.387+0000 7fe7f5ffb700 1 -- 172.21.15.103:0/776573578 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fe7f002c020 con 0x7fe800132f50 2023-07-02T09:03:59.388 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:03:59.390 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe7e40377c0 msgr2=0x7fe7e4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:59.390 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 --2- 172.21.15.103:0/776573578 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe7e40377c0 0x7fe7e4039cd0 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:59.390 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800132f50 msgr2=0x7fe8001ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:03:59.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 --2- 172.21.15.103:0/776573578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800132f50 0x7fe8001ba9b0 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:59.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 shutdown_connections 2023-07-02T09:03:59.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 --2- 172.21.15.103:0/776573578 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe7e40377c0 0x7fe7e4039cd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:59.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 --2- 172.21.15.103:0/776573578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe800132f50 0x7fe8001ba9b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:03:59.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 >> 172.21.15.103:0/776573578 conn(0x7fe8000a2080 msgr2=0x7fe8000a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:03:59.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 shutdown_connections 2023-07-02T09:03:59.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:03:59.390+0000 7fe80786b700 1 -- 172.21.15.103:0/776573578 wait complete. 2023-07-02T09:03:59.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:00.404 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:00.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.536+0000 7f04f98e2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:00.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.541+0000 7f04f98e2700 1 Processor -- start 2023-07-02T09:04:00.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.541+0000 7f04f98e2700 1 -- start start 2023-07-02T09:04:00.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.542+0000 7f04f98e2700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f04f4132030 0x7f04f4134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:00.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.542+0000 7f04f98e2700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f406df60 con 0x7f04f412f250 2023-07-02T09:04:00.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.542+0000 7f04f98e2700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f409a710 con 0x7f04f4132030 2023-07-02T09:04:00.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.542+0000 7f04f27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f04f4132030 0x7f04f4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:00.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.542+0000 7f04f2ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f04f412f250 0x7f04f412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57282/0 (socket says 172.21.15.103:57282) 2023-07-02T09:04:00.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.542+0000 7f04f27fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f04f4132030 0x7f04f4134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58414/0 (socket says 172.21.15.103:58414) 2023-07-02T09:04:00.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.542+0000 7f04f2ffd700 1 -- 172.21.15.103:0/2432759038 learned_addr learned my addr 172.21.15.103:0/2432759038 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:00.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f27fc700 1 -- 172.21.15.103:0/2432759038 >> v1:172.21.15.103:6789/0 conn(0x7f04f412f250 legacy=0x7f04f412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:00.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f27fc700 1 -- 172.21.15.103:0/2432759038 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f412ed20 con 0x7f04f4132030 2023-07-02T09:04:00.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f1ffb700 1 -- 172.21.15.103:0/2432759038 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830626897 0 0) 0x7f04f406df60 con 0x7f04f412f250 2023-07-02T09:04:00.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f27fc700 1 --2- 172.21.15.103:0/2432759038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f4132030 0x7f04f4134450 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=762a8fc86fe2eae9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:00.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f1ffb700 1 -- 172.21.15.103:0/2432759038 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f04e4009cd0 con 0x7f04f4132030 2023-07-02T09:04:00.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f1ffb700 1 -- 172.21.15.103:0/2432759038 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f04e4009e30 con 0x7f04f4132030 2023-07-02T09:04:00.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f1ffb700 1 -- 172.21.15.103:0/2432759038 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f04e400e360 con 0x7f04f4132030 2023-07-02T09:04:00.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f98e2700 1 -- 172.21.15.103:0/2432759038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f4132030 msgr2=0x7f04f4134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:00.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f98e2700 1 --2- 172.21.15.103:0/2432759038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f4132030 0x7f04f4134450 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:00.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f98e2700 1 -- 172.21.15.103:0/2432759038 shutdown_connections 2023-07-02T09:04:00.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f98e2700 1 --2- 172.21.15.103:0/2432759038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f4132030 0x7f04f4134450 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:00.546 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.543+0000 7f04f98e2700 1 -- 172.21.15.103:0/2432759038 >> 172.21.15.103:0/2432759038 conn(0x7f04f409d810 msgr2=0x7f04f409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:00.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.544+0000 7f04f98e2700 1 -- 172.21.15.103:0/2432759038 shutdown_connections 2023-07-02T09:04:00.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.544+0000 7f04f98e2700 1 -- 172.21.15.103:0/2432759038 wait complete. 2023-07-02T09:04:00.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.544+0000 7f04f98e2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:00.548 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.545+0000 7f04f98e2700 1 Processor -- start 2023-07-02T09:04:00.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.545+0000 7f04f98e2700 1 -- start start 2023-07-02T09:04:00.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.545+0000 7f04f98e2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f412f250 0x7f04f40710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:00.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04f98e2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f411dbb0 con 0x7f04f412f250 2023-07-02T09:04:00.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04f2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f412f250 0x7f04f40710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:00.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04f2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f412f250 0x7f04f40710e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58428/0 (socket says 172.21.15.103:58428) 2023-07-02T09:04:00.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04f2ffd700 1 -- 172.21.15.103:0/2999582086 learned_addr learned my addr 172.21.15.103:0/2999582086 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:00.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04f2ffd700 1 -- 172.21.15.103:0/2999582086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f411ca90 con 0x7f04f412f250 2023-07-02T09:04:00.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04f2ffd700 1 --2- 172.21.15.103:0/2999582086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f412f250 0x7f04f40710e0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:00.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04dbfff700 1 -- 172.21.15.103:0/2999582086 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f04e8004b80 con 0x7f04f412f250 2023-07-02T09:04:00.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04dbfff700 1 -- 172.21.15.103:0/2999582086 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f04e8004ce0 con 0x7f04f412f250 2023-07-02T09:04:00.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f411d880 con 0x7f04f412f250 2023-07-02T09:04:00.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04dbfff700 1 -- 172.21.15.103:0/2999582086 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f04e800c4f0 con 0x7f04f412f250 2023-07-02T09:04:00.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.546+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f411d070 con 0x7f04f412f250 2023-07-02T09:04:00.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.547+0000 7f04dbfff700 1 -- 172.21.15.103:0/2999582086 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f04e800c650 con 0x7f04f412f250 2023-07-02T09:04:00.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.547+0000 7f04dbfff700 1 --2- 172.21.15.103:0/2999582086 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04e00377c0 0x7f04e0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:00.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.547+0000 7f04dbfff700 1 -- 172.21.15.103:0/2999582086 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f04e8046990 con 0x7f04f412f250 2023-07-02T09:04:00.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.547+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f411da10 con 0x7f04f412f250 2023-07-02T09:04:00.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.547+0000 7f04f27fc700 1 --2- 172.21.15.103:0/2999582086 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04e00377c0 0x7f04e0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:00.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.548+0000 7f04f27fc700 1 --2- 172.21.15.103:0/2999582086 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04e00377c0 0x7f04e0039cd0 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:00.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.552+0000 7f04dbfff700 1 -- 172.21.15.103:0/2999582086 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f04f411da10 con 0x7f04f412f250 2023-07-02T09:04:00.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.740+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f04f411da10 con 0x7f04f412f250 2023-07-02T09:04:00.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.741+0000 7f04dbfff700 1 -- 172.21.15.103:0/2999582086 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f04f411da10 con 0x7f04f412f250 2023-07-02T09:04:00.741 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:00.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04e00377c0 msgr2=0x7f04e0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:00.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 --2- 172.21.15.103:0/2999582086 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04e00377c0 0x7f04e0039cd0 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:00.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f412f250 msgr2=0x7f04f40710e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:00.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 --2- 172.21.15.103:0/2999582086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f412f250 0x7f04f40710e0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:00.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 shutdown_connections 2023-07-02T09:04:00.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 --2- 172.21.15.103:0/2999582086 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04e00377c0 0x7f04e0039cd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:00.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 --2- 172.21.15.103:0/2999582086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04f412f250 0x7f04f40710e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:00.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 >> 172.21.15.103:0/2999582086 conn(0x7f04f409d810 msgr2=0x7f04f4131700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:00.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 shutdown_connections 2023-07-02T09:04:00.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:00.743+0000 7f04f98e2700 1 -- 172.21.15.103:0/2999582086 wait complete. 2023-07-02T09:04:00.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:01.756 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:01.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.900+0000 7fc8bf568700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:01.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.905+0000 7fc8bf568700 1 Processor -- start 2023-07-02T09:04:01.905 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.905+0000 7fc8bf568700 1 -- start start 2023-07-02T09:04:01.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.906+0000 7fc8bf568700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc8b8099520 0x7fc8b8135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:01.906 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.906+0000 7fc8bf568700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b806f8b0 con 0x7fc8b8098a00 2023-07-02T09:04:01.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.906+0000 7fc8bf568700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b8072510 con 0x7fc8b8099520 2023-07-02T09:04:01.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.906+0000 7fc8bcb03700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc8b8099520 0x7fc8b8135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:01.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.906+0000 7fc8bd304700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc8b8098a00 0x7fc8b8098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57284/0 (socket says 172.21.15.103:57284) 2023-07-02T09:04:01.907 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.906+0000 7fc8bd304700 1 -- 172.21.15.103:0/3240220242 learned_addr learned my addr 172.21.15.103:0/3240220242 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:01.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.906+0000 7fc8bcb03700 1 -- 172.21.15.103:0/3240220242 >> v1:172.21.15.103:6789/0 conn(0x7fc8b8098a00 legacy=0x7fc8b8098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:01.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.906+0000 7fc8bcb03700 1 -- 172.21.15.103:0/3240220242 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b809aec0 con 0x7fc8b8099520 2023-07-02T09:04:01.908 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8bcb03700 1 --2- 172.21.15.103:0/3240220242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8099520 0x7fc8b8135ca0 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50efab778a0710a9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:01.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8affff700 1 -- 172.21.15.103:0/3240220242 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc8b4009cd0 con 0x7fc8b8099520 2023-07-02T09:04:01.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8affff700 1 -- 172.21.15.103:0/3240220242 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc8b4009e30 con 0x7fc8b8099520 2023-07-02T09:04:01.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8affff700 1 -- 172.21.15.103:0/3240220242 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc8b400e360 con 0x7fc8b8099520 2023-07-02T09:04:01.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8bf568700 1 -- 172.21.15.103:0/3240220242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8099520 msgr2=0x7fc8b8135ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:01.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8bf568700 1 --2- 172.21.15.103:0/3240220242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8099520 0x7fc8b8135ca0 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:01.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8bf568700 1 -- 172.21.15.103:0/3240220242 shutdown_connections 2023-07-02T09:04:01.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8bf568700 1 --2- 172.21.15.103:0/3240220242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8099520 0x7fc8b8135ca0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:01.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.907+0000 7fc8bf568700 1 -- 172.21.15.103:0/3240220242 >> 172.21.15.103:0/3240220242 conn(0x7fc8b809d810 msgr2=0x7fc8b809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:01.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.908+0000 7fc8bf568700 1 -- 172.21.15.103:0/3240220242 shutdown_connections 2023-07-02T09:04:01.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.908+0000 7fc8bf568700 1 -- 172.21.15.103:0/3240220242 wait complete. 2023-07-02T09:04:01.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.908+0000 7fc8bf568700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:01.911 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.909+0000 7fc8bf568700 1 Processor -- start 2023-07-02T09:04:01.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.909+0000 7fc8bf568700 1 -- start start 2023-07-02T09:04:01.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.909+0000 7fc8bf568700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8098a00 0x7fc8b81c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:01.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.909+0000 7fc8bf568700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b806d4f0 con 0x7fc8b8098a00 2023-07-02T09:04:01.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.909+0000 7fc8bd304700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8098a00 0x7fc8b81c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:01.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.909+0000 7fc8bd304700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8098a00 0x7fc8b81c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58436/0 (socket says 172.21.15.103:58436) 2023-07-02T09:04:01.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.909+0000 7fc8bd304700 1 -- 172.21.15.103:0/774820754 learned_addr learned my addr 172.21.15.103:0/774820754 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:01.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.910+0000 7fc8bd304700 1 -- 172.21.15.103:0/774820754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b806ed10 con 0x7fc8b8098a00 2023-07-02T09:04:01.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.910+0000 7fc8bd304700 1 --2- 172.21.15.103:0/774820754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8098a00 0x7fc8b81c6c00 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:01.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.910+0000 7fc8adffb700 1 -- 172.21.15.103:0/774820754 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc8a8005a60 con 0x7fc8b8098a00 2023-07-02T09:04:01.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.910+0000 7fc8adffb700 1 -- 172.21.15.103:0/774820754 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc8a8005bc0 con 0x7fc8b8098a00 2023-07-02T09:04:01.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.910+0000 7fc8adffb700 1 -- 172.21.15.103:0/774820754 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc8a8005e60 con 0x7fc8b8098a00 2023-07-02T09:04:01.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.910+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b806d150 con 0x7fc8b8098a00 2023-07-02T09:04:01.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.910+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b8122770 con 0x7fc8b8098a00 2023-07-02T09:04:01.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.911+0000 7fc8adffb700 1 -- 172.21.15.103:0/774820754 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fc8a800c7b0 con 0x7fc8b8098a00 2023-07-02T09:04:01.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.911+0000 7fc8adffb700 1 --2- 172.21.15.103:0/774820754 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc89c0377c0 0x7fc89c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:01.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.911+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b806d340 con 0x7fc8b8098a00 2023-07-02T09:04:01.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.911+0000 7fc8adffb700 1 -- 172.21.15.103:0/774820754 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc8a8046bf0 con 0x7fc8b8098a00 2023-07-02T09:04:01.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.912+0000 7fc8bcb03700 1 --2- 172.21.15.103:0/774820754 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc89c0377c0 0x7fc89c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:01.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.912+0000 7fc8bcb03700 1 --2- 172.21.15.103:0/774820754 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc89c0377c0 0x7fc89c039cd0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:01.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:01.916+0000 7fc8adffb700 1 -- 172.21.15.103:0/774820754 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc8a800a4a0 con 0x7fc8b8098a00 2023-07-02T09:04:02.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.110+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc8b8061ea0 con 0x7fc8b8098a00 2023-07-02T09:04:02.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.110+0000 7fc8adffb700 1 -- 172.21.15.103:0/774820754 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fc8a800a4a0 con 0x7fc8b8098a00 2023-07-02T09:04:02.110 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:02.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc89c0377c0 msgr2=0x7fc89c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 --2- 172.21.15.103:0/774820754 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc89c0377c0 0x7fc89c039cd0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8098a00 msgr2=0x7fc8b81c6c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 --2- 172.21.15.103:0/774820754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8098a00 0x7fc8b81c6c00 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 shutdown_connections 2023-07-02T09:04:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 --2- 172.21.15.103:0/774820754 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc89c0377c0 0x7fc89c039cd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 --2- 172.21.15.103:0/774820754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b8098a00 0x7fc8b81c6c00 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 >> 172.21.15.103:0/774820754 conn(0x7fc8b809d810 msgr2=0x7fc8b81306f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:02.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 shutdown_connections 2023-07-02T09:04:02.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:02.112+0000 7fc8bf568700 1 -- 172.21.15.103:0/774820754 wait complete. 2023-07-02T09:04:02.125 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:03.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:03.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.258+0000 7fd3febca700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:03.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.264+0000 7fd3febca700 1 Processor -- start 2023-07-02T09:04:03.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.264+0000 7fd3febca700 1 -- start start 2023-07-02T09:04:03.265 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.264+0000 7fd3febca700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd3f812f250 0x7fd3f812f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.264+0000 7fd3febca700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f806df60 con 0x7fd3f8098a00 2023-07-02T09:04:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.264+0000 7fd3febca700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f809a710 con 0x7fd3f812f250 2023-07-02T09:04:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3fc966700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd3f812f250 0x7fd3f812f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3f7fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd3f8098a00 0x7fd3f8098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:57292/0 (socket says 172.21.15.103:57292) 2023-07-02T09:04:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3f7fff700 1 -- 172.21.15.103:0/2185116697 learned_addr learned my addr 172.21.15.103:0/2185116697 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3fc966700 1 -- 172.21.15.103:0/2185116697 >> v1:172.21.15.103:6789/0 conn(0x7fd3f8098a00 legacy=0x7fd3f8098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3fc966700 1 -- 172.21.15.103:0/2185116697 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f812ed20 con 0x7fd3f812f250 2023-07-02T09:04:03.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3fc966700 1 --2- 172.21.15.103:0/2185116697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f812f250 0x7fd3f812f660 crc :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60fe989e51b261d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:03.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3f77fe700 1 -- 172.21.15.103:0/2185116697 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd3ec009cd0 con 0x7fd3f812f250 2023-07-02T09:04:03.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3f77fe700 1 -- 172.21.15.103:0/2185116697 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd3ec009e30 con 0x7fd3f812f250 2023-07-02T09:04:03.267 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.265+0000 7fd3f77fe700 1 -- 172.21.15.103:0/2185116697 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd3ec00e360 con 0x7fd3f812f250 2023-07-02T09:04:03.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.266+0000 7fd3febca700 1 -- 172.21.15.103:0/2185116697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f812f250 msgr2=0x7fd3f812f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:03.269 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.266+0000 7fd3febca700 1 --2- 172.21.15.103:0/2185116697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f812f250 0x7fd3f812f660 crc :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:03.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.266+0000 7fd3febca700 1 -- 172.21.15.103:0/2185116697 shutdown_connections 2023-07-02T09:04:03.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.266+0000 7fd3febca700 1 --2- 172.21.15.103:0/2185116697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f812f250 0x7fd3f812f660 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:03.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.266+0000 7fd3febca700 1 -- 172.21.15.103:0/2185116697 >> 172.21.15.103:0/2185116697 conn(0x7fd3f809d810 msgr2=0x7fd3f809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:03.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.266+0000 7fd3febca700 1 -- 172.21.15.103:0/2185116697 shutdown_connections 2023-07-02T09:04:03.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.266+0000 7fd3febca700 1 -- 172.21.15.103:0/2185116697 wait complete. 2023-07-02T09:04:03.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.266+0000 7fd3febca700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:03.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.267+0000 7fd3febca700 1 Processor -- start 2023-07-02T09:04:03.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.267+0000 7fd3febca700 1 -- start start 2023-07-02T09:04:03.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.268+0000 7fd3febca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f8098a00 0x7fd3f8070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:03.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.268+0000 7fd3febca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f8072080 con 0x7fd3f8098a00 2023-07-02T09:04:03.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.268+0000 7fd3fc966700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f8098a00 0x7fd3f8070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:03.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.268+0000 7fd3fc966700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f8098a00 0x7fd3f8070080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58448/0 (socket says 172.21.15.103:58448) 2023-07-02T09:04:03.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.268+0000 7fd3fc966700 1 -- 172.21.15.103:0/2774457143 learned_addr learned my addr 172.21.15.103:0/2774457143 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:03.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.269+0000 7fd3fc966700 1 -- 172.21.15.103:0/2774457143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f8071e10 con 0x7fd3f8098a00 2023-07-02T09:04:03.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.269+0000 7fd3fc966700 1 --2- 172.21.15.103:0/2774457143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f8098a00 0x7fd3f8070080 crc :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:03.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.269+0000 7fd3f57fa700 1 -- 172.21.15.103:0/2774457143 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd3ec013530 con 0x7fd3f8098a00 2023-07-02T09:04:03.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.269+0000 7fd3f57fa700 1 -- 172.21.15.103:0/2774457143 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd3ec013690 con 0x7fd3f8098a00 2023-07-02T09:04:03.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.269+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f806fc60 con 0x7fd3f8098a00 2023-07-02T09:04:03.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.269+0000 7fd3f57fa700 1 -- 172.21.15.103:0/2774457143 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd3ec013930 con 0x7fd3f8098a00 2023-07-02T09:04:03.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.269+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f8117be0 con 0x7fd3f8098a00 2023-07-02T09:04:03.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.270+0000 7fd3f57fa700 1 -- 172.21.15.103:0/2774457143 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd3ec013ae0 con 0x7fd3f8098a00 2023-07-02T09:04:03.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.270+0000 7fd3f57fa700 1 --2- 172.21.15.103:0/2774457143 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd3dc0377c0 0x7fd3dc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:03.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.270+0000 7fd3f57fa700 1 -- 172.21.15.103:0/2774457143 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd3ec018040 con 0x7fd3f8098a00 2023-07-02T09:04:03.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.270+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f8061ea0 con 0x7fd3f8098a00 2023-07-02T09:04:03.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.270+0000 7fd3f7fff700 1 --2- 172.21.15.103:0/2774457143 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd3dc0377c0 0x7fd3dc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:03.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.271+0000 7fd3f7fff700 1 --2- 172.21.15.103:0/2774457143 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd3dc0377c0 0x7fd3dc039cd0 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:03.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.275+0000 7fd3f57fa700 1 -- 172.21.15.103:0/2774457143 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd3ec018e00 con 0x7fd3f8098a00 2023-07-02T09:04:03.464 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.464+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd3f8073ca0 con 0x7fd3f8098a00 2023-07-02T09:04:03.465 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.465+0000 7fd3f57fa700 1 -- 172.21.15.103:0/2774457143 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd3ec010e60 con 0x7fd3f8098a00 2023-07-02T09:04:03.465 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:03.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd3dc0377c0 msgr2=0x7fd3dc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:03.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 --2- 172.21.15.103:0/2774457143 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd3dc0377c0 0x7fd3dc039cd0 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:03.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f8098a00 msgr2=0x7fd3f8070080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:03.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 --2- 172.21.15.103:0/2774457143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f8098a00 0x7fd3f8070080 crc :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:03.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 shutdown_connections 2023-07-02T09:04:03.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 --2- 172.21.15.103:0/2774457143 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd3dc0377c0 0x7fd3dc039cd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:03.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 --2- 172.21.15.103:0/2774457143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3f8098a00 0x7fd3f8070080 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:03.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 >> 172.21.15.103:0/2774457143 conn(0x7fd3f809d810 msgr2=0x7fd3f81b6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:03.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 shutdown_connections 2023-07-02T09:04:03.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:03.467+0000 7fd3febca700 1 -- 172.21.15.103:0/2774457143 wait complete. 2023-07-02T09:04:03.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:04.481 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:04.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~259s 2023-07-02T09:04:04.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.621+0000 7f31390ff700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:04.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.626+0000 7f31390ff700 1 Processor -- start 2023-07-02T09:04:04.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.626+0000 7f31390ff700 1 -- start start 2023-07-02T09:04:04.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.626+0000 7f31390ff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f313409a220 0x7f31341348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:04.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.626+0000 7f31390ff700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313409cea0 con 0x7f313406d6e0 2023-07-02T09:04:04.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.626+0000 7f31390ff700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313406d490 con 0x7f313409a220 2023-07-02T09:04:04.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f313259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f313409a220 0x7f31341348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:04.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f313259c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f313409a220 0x7f31341348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50414/0 (socket says 172.21.15.103:50414) 2023-07-02T09:04:04.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f3132d9d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f313406d6e0 0x7f3134099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48582/0 (socket says 172.21.15.103:48582) 2023-07-02T09:04:04.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f313259c700 1 -- 172.21.15.103:0/986383599 learned_addr learned my addr 172.21.15.103:0/986383599 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:04.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f313259c700 1 -- 172.21.15.103:0/986383599 >> v1:172.21.15.103:6789/0 conn(0x7f313406d6e0 legacy=0x7f3134099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:04.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f313259c700 1 -- 172.21.15.103:0/986383599 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313406dc80 con 0x7f313409a220 2023-07-02T09:04:04.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f3131d9b700 1 -- 172.21.15.103:0/986383599 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144538077 0 0) 0x7f313409cea0 con 0x7f313406d6e0 2023-07-02T09:04:04.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f313259c700 1 --2- 172.21.15.103:0/986383599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313409a220 0x7f31341348e0 crc :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9fc0e7e5e5d042 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:04.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f3131d9b700 1 -- 172.21.15.103:0/986383599 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3128009d90 con 0x7f313409a220 2023-07-02T09:04:04.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f3131d9b700 1 -- 172.21.15.103:0/986383599 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f312800e230 con 0x7f313409a220 2023-07-02T09:04:04.629 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.627+0000 7f3131d9b700 1 -- 172.21.15.103:0/986383599 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f312800e4d0 con 0x7f313409a220 2023-07-02T09:04:04.630 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.628+0000 7f31390ff700 1 -- 172.21.15.103:0/986383599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313409a220 msgr2=0x7f31341348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:04.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.628+0000 7f31390ff700 1 --2- 172.21.15.103:0/986383599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313409a220 0x7f31341348e0 crc :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:04.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.628+0000 7f31390ff700 1 -- 172.21.15.103:0/986383599 shutdown_connections 2023-07-02T09:04:04.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.628+0000 7f31390ff700 1 --2- 172.21.15.103:0/986383599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313409a220 0x7f31341348e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:04.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.628+0000 7f31390ff700 1 -- 172.21.15.103:0/986383599 >> 172.21.15.103:0/986383599 conn(0x7f313409d810 msgr2=0x7f313409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:04.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.628+0000 7f31390ff700 1 -- 172.21.15.103:0/986383599 shutdown_connections 2023-07-02T09:04:04.631 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.628+0000 7f31390ff700 1 -- 172.21.15.103:0/986383599 wait complete. 2023-07-02T09:04:04.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.628+0000 7f31390ff700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:04.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.629+0000 7f31390ff700 1 Processor -- start 2023-07-02T09:04:04.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.629+0000 7f31390ff700 1 -- start start 2023-07-02T09:04:04.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.630+0000 7f31390ff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313406d6e0 0x7f313412ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:04.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.630+0000 7f31390ff700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313411db90 con 0x7f313406d6e0 2023-07-02T09:04:04.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.630+0000 7f3132d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313406d6e0 0x7f313412ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:04.632 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.630+0000 7f3132d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313406d6e0 0x7f313412ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50430/0 (socket says 172.21.15.103:50430) 2023-07-02T09:04:04.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.630+0000 7f3132d9d700 1 -- 172.21.15.103:0/3380650872 learned_addr learned my addr 172.21.15.103:0/3380650872 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:04.633 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.631+0000 7f3132d9d700 1 -- 172.21.15.103:0/3380650872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313411e370 con 0x7f313406d6e0 2023-07-02T09:04:04.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.631+0000 7f3132d9d700 1 --2- 172.21.15.103:0/3380650872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313406d6e0 0x7f313412ded0 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:04.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.631+0000 7f311b7fe700 1 -- 172.21.15.103:0/3380650872 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f3124004b80 con 0x7f313406d6e0 2023-07-02T09:04:04.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.631+0000 7f311b7fe700 1 -- 172.21.15.103:0/3380650872 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f3124004ce0 con 0x7f313406d6e0 2023-07-02T09:04:04.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.631+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313411d880 con 0x7f313406d6e0 2023-07-02T09:04:04.634 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.631+0000 7f311b7fe700 1 -- 172.21.15.103:0/3380650872 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f312400c550 con 0x7f313406d6e0 2023-07-02T09:04:04.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.631+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31341203e0 con 0x7f313406d6e0 2023-07-02T09:04:04.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.632+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3134061ea0 con 0x7f313406d6e0 2023-07-02T09:04:04.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.632+0000 7f311b7fe700 1 -- 172.21.15.103:0/3380650872 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f312400c6b0 con 0x7f313406d6e0 2023-07-02T09:04:04.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.632+0000 7f311b7fe700 1 --2- 172.21.15.103:0/3380650872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31200377c0 0x7f3120039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:04.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.632+0000 7f311b7fe700 1 -- 172.21.15.103:0/3380650872 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f3124046b90 con 0x7f313406d6e0 2023-07-02T09:04:04.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.633+0000 7f313259c700 1 --2- 172.21.15.103:0/3380650872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31200377c0 0x7f3120039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:04.635 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.633+0000 7f313259c700 1 --2- 172.21.15.103:0/3380650872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31200377c0 0x7f3120039cd0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:04.637 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.637+0000 7f311b7fe700 1 -- 172.21.15.103:0/3380650872 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f312400ad20 con 0x7f313406d6e0 2023-07-02T09:04:04.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.827+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3134073ca0 con 0x7f313406d6e0 2023-07-02T09:04:04.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.827+0000 7f311b7fe700 1 -- 172.21.15.103:0/3380650872 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f312400a910 con 0x7f313406d6e0 2023-07-02T09:04:04.828 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.829+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31200377c0 msgr2=0x7f3120039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.829+0000 7f31390ff700 1 --2- 172.21.15.103:0/3380650872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31200377c0 0x7f3120039cd0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.830+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313406d6e0 msgr2=0x7f313412ded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.830+0000 7f31390ff700 1 --2- 172.21.15.103:0/3380650872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313406d6e0 0x7f313412ded0 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.830+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 shutdown_connections 2023-07-02T09:04:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.830+0000 7f31390ff700 1 --2- 172.21.15.103:0/3380650872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31200377c0 0x7f3120039cd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.830+0000 7f31390ff700 1 --2- 172.21.15.103:0/3380650872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f313406d6e0 0x7f313412ded0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.830+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 >> 172.21.15.103:0/3380650872 conn(0x7f313409d810 msgr2=0x7f3134070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.830+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 shutdown_connections 2023-07-02T09:04:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:04.830+0000 7f31390ff700 1 -- 172.21.15.103:0/3380650872 wait complete. 2023-07-02T09:04:04.843 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:05.844 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:05.986 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.982+0000 7f1d0ebe7700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:05.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.987+0000 7f1d0ebe7700 1 Processor -- start 2023-07-02T09:04:05.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.987+0000 7f1d0ebe7700 1 -- start start 2023-07-02T09:04:05.988 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d0ebe7700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d0806d6e0 0x7f1d08099b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:05.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d0ebe7700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0809cea0 con 0x7f1d0809a050 2023-07-02T09:04:05.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d0ebe7700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0806d490 con 0x7f1d0806d6e0 2023-07-02T09:04:05.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d0c983700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d0806d6e0 0x7f1d08099b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:05.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d07fff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1d0809a050 0x7f1d0812f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48588/0 (socket says 172.21.15.103:48588) 2023-07-02T09:04:05.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d0c983700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1d0806d6e0 0x7f1d08099b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50432/0 (socket says 172.21.15.103:50432) 2023-07-02T09:04:05.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d07fff700 1 -- 172.21.15.103:0/1552368219 learned_addr learned my addr 172.21.15.103:0/1552368219 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d0c983700 1 -- 172.21.15.103:0/1552368219 >> v1:172.21.15.103:6789/0 conn(0x7f1d0809a050 legacy=0x7f1d0812f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d0c983700 1 -- 172.21.15.103:0/1552368219 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0806dc80 con 0x7f1d0806d6e0 2023-07-02T09:04:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d077fe700 1 -- 172.21.15.103:0/1552368219 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407464850 0 0) 0x7f1d0809cea0 con 0x7f1d0809a050 2023-07-02T09:04:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d0c983700 1 --2- 172.21.15.103:0/1552368219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d08099b10 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6cf625b574abe31 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d077fe700 1 -- 172.21.15.103:0/1552368219 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1cfc009cd0 con 0x7f1d0806d6e0 2023-07-02T09:04:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d077fe700 1 -- 172.21.15.103:0/1552368219 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1cfc009e30 con 0x7f1d0806d6e0 2023-07-02T09:04:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.988+0000 7f1d077fe700 1 -- 172.21.15.103:0/1552368219 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1cfc00e360 con 0x7f1d0806d6e0 2023-07-02T09:04:05.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.989+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/1552368219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 msgr2=0x7f1d08099b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:05.992 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.989+0000 7f1d0ebe7700 1 --2- 172.21.15.103:0/1552368219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d08099b10 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:05.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.989+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/1552368219 shutdown_connections 2023-07-02T09:04:05.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.989+0000 7f1d0ebe7700 1 --2- 172.21.15.103:0/1552368219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d08099b10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:05.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.989+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/1552368219 >> 172.21.15.103:0/1552368219 conn(0x7f1d0809d810 msgr2=0x7f1d0809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:05.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.989+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/1552368219 shutdown_connections 2023-07-02T09:04:05.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.989+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/1552368219 wait complete. 2023-07-02T09:04:05.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.989+0000 7f1d0ebe7700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:05.993 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.990+0000 7f1d0ebe7700 1 Processor -- start 2023-07-02T09:04:05.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.990+0000 7f1d0ebe7700 1 -- start start 2023-07-02T09:04:05.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.991+0000 7f1d0ebe7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d081c48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:05.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.991+0000 7f1d0ebe7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08121d60 con 0x7f1d0806d6e0 2023-07-02T09:04:05.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.991+0000 7f1d0c983700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d081c48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:05.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.991+0000 7f1d0c983700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d081c48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50436/0 (socket says 172.21.15.103:50436) 2023-07-02T09:04:05.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.991+0000 7f1d0c983700 1 -- 172.21.15.103:0/4281196872 learned_addr learned my addr 172.21.15.103:0/4281196872 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:05.994 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.991+0000 7f1d0c983700 1 -- 172.21.15.103:0/4281196872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08121ac0 con 0x7f1d0806d6e0 2023-07-02T09:04:05.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.992+0000 7f1d0c983700 1 --2- 172.21.15.103:0/4281196872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d081c48b0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:05.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.992+0000 7f1d057fa700 1 -- 172.21.15.103:0/4281196872 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1cfc013530 con 0x7f1d0806d6e0 2023-07-02T09:04:05.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.992+0000 7f1d057fa700 1 -- 172.21.15.103:0/4281196872 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1cfc013690 con 0x7f1d0806d6e0 2023-07-02T09:04:05.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.992+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d08122930 con 0x7f1d0806d6e0 2023-07-02T09:04:05.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.992+0000 7f1d057fa700 1 -- 172.21.15.103:0/4281196872 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1cfc013930 con 0x7f1d0806d6e0 2023-07-02T09:04:05.995 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.992+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d081224d0 con 0x7f1d0806d6e0 2023-07-02T09:04:05.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.993+0000 7f1d057fa700 1 -- 172.21.15.103:0/4281196872 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f1cfc013ae0 con 0x7f1d0806d6e0 2023-07-02T09:04:05.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.993+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d08061ea0 con 0x7f1d0806d6e0 2023-07-02T09:04:05.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.993+0000 7f1d057fa700 1 --2- 172.21.15.103:0/4281196872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cec0377b0 0x7f1cec039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:05.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.993+0000 7f1d057fa700 1 -- 172.21.15.103:0/4281196872 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f1cfc018040 con 0x7f1d0806d6e0 2023-07-02T09:04:05.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.993+0000 7f1d07fff700 1 --2- 172.21.15.103:0/4281196872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cec0377b0 0x7f1cec039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:05.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.994+0000 7f1d07fff700 1 --2- 172.21.15.103:0/4281196872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cec0377b0 0x7f1cec039cc0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:05.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:05.998+0000 7f1d057fa700 1 -- 172.21.15.103:0/4281196872 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f1cfc018e00 con 0x7f1d0806d6e0 2023-07-02T09:04:06.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.187+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1d08073ca0 con 0x7f1d0806d6e0 2023-07-02T09:04:06.188 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.187+0000 7f1d057fa700 1 -- 172.21.15.103:0/4281196872 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f1cfc0189f0 con 0x7f1d0806d6e0 2023-07-02T09:04:06.188 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:06.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cec0377b0 msgr2=0x7f1cec039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:06.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 --2- 172.21.15.103:0/4281196872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cec0377b0 0x7f1cec039cc0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:06.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 msgr2=0x7f1d081c48b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:06.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 --2- 172.21.15.103:0/4281196872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d081c48b0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:06.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 shutdown_connections 2023-07-02T09:04:06.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 --2- 172.21.15.103:0/4281196872 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1cec0377b0 0x7f1cec039cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:06.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 --2- 172.21.15.103:0/4281196872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0806d6e0 0x7f1d081c48b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:06.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 >> 172.21.15.103:0/4281196872 conn(0x7f1d0809d810 msgr2=0x7f1d08117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:06.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 shutdown_connections 2023-07-02T09:04:06.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:06.190+0000 7f1d0ebe7700 1 -- 172.21.15.103:0/4281196872 wait complete. 2023-07-02T09:04:06.203 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:07.203 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:07.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.343+0000 7f0896431700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:07.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.348+0000 7f0896431700 1 Processor -- start 2023-07-02T09:04:07.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.348+0000 7f0896431700 1 -- start start 2023-07-02T09:04:07.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f0896431700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0890132f50 0x7f0890133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:07.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f0896431700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089009e260 con 0x7f089009c580 2023-07-02T09:04:07.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f0896431700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08900a1640 con 0x7f0890132f50 2023-07-02T09:04:07.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f088ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0890132f50 0x7f0890133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:07.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f088f7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f089009c580 0x7f089009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48600/0 (socket says 172.21.15.103:48600) 2023-07-02T09:04:07.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f088f7fe700 1 -- 172.21.15.103:0/2297035179 learned_addr learned my addr 172.21.15.103:0/2297035179 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:07.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f088ffff700 1 -- 172.21.15.103:0/2297035179 >> v1:172.21.15.103:6789/0 conn(0x7f089009c580 legacy=0x7f089009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:07.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f088ffff700 1 -- 172.21.15.103:0/2297035179 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0890132a20 con 0x7f0890132f50 2023-07-02T09:04:07.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f088effd700 1 -- 172.21.15.103:0/2297035179 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235790609 0 0) 0x7f089009e260 con 0x7f089009c580 2023-07-02T09:04:07.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f088ffff700 1 --2- 172.21.15.103:0/2297035179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0890132f50 0x7f0890133360 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909b97106b497c25 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:07.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.349+0000 7f088effd700 1 -- 172.21.15.103:0/2297035179 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0884009cd0 con 0x7f0890132f50 2023-07-02T09:04:07.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f088effd700 1 -- 172.21.15.103:0/2297035179 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0884009e30 con 0x7f0890132f50 2023-07-02T09:04:07.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f088effd700 1 -- 172.21.15.103:0/2297035179 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f088400e360 con 0x7f0890132f50 2023-07-02T09:04:07.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f0896431700 1 -- 172.21.15.103:0/2297035179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0890132f50 msgr2=0x7f0890133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:07.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f0896431700 1 --2- 172.21.15.103:0/2297035179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0890132f50 0x7f0890133360 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:07.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f0896431700 1 -- 172.21.15.103:0/2297035179 shutdown_connections 2023-07-02T09:04:07.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f0896431700 1 --2- 172.21.15.103:0/2297035179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0890132f50 0x7f0890133360 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:07.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f0896431700 1 -- 172.21.15.103:0/2297035179 >> 172.21.15.103:0/2297035179 conn(0x7f08900a2080 msgr2=0x7f08900a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:07.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f0896431700 1 -- 172.21.15.103:0/2297035179 shutdown_connections 2023-07-02T09:04:07.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.350+0000 7f0896431700 1 -- 172.21.15.103:0/2297035179 wait complete. 2023-07-02T09:04:07.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.351+0000 7f0896431700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:07.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.352+0000 7f0896431700 1 Processor -- start 2023-07-02T09:04:07.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.352+0000 7f0896431700 1 -- start start 2023-07-02T09:04:07.355 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f0896431700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f089009c580 0x7f08901baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:07.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f0896431700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08901ba610 con 0x7f089009c580 2023-07-02T09:04:07.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f088ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f089009c580 0x7f08901baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:07.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f088ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f089009c580 0x7f08901baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50454/0 (socket says 172.21.15.103:50454) 2023-07-02T09:04:07.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f088ffff700 1 -- 172.21.15.103:0/2646523568 learned_addr learned my addr 172.21.15.103:0/2646523568 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:07.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f088ffff700 1 -- 172.21.15.103:0/2646523568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089006dfc0 con 0x7f089009c580 2023-07-02T09:04:07.356 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f088ffff700 1 --2- 172.21.15.103:0/2646523568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f089009c580 0x7f08901baa60 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:07.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f088cff9700 1 -- 172.21.15.103:0/2646523568 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0884013530 con 0x7f089009c580 2023-07-02T09:04:07.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f088cff9700 1 -- 172.21.15.103:0/2646523568 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0884013690 con 0x7f089009c580 2023-07-02T09:04:07.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f088cff9700 1 -- 172.21.15.103:0/2646523568 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0884013930 con 0x7f089009c580 2023-07-02T09:04:07.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.353+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08901ba270 con 0x7f089009c580 2023-07-02T09:04:07.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.354+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089006dba0 con 0x7f089009c580 2023-07-02T09:04:07.357 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.355+0000 7f088cff9700 1 -- 172.21.15.103:0/2646523568 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f0884013ae0 con 0x7f089009c580 2023-07-02T09:04:07.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.355+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08901ba460 con 0x7f089009c580 2023-07-02T09:04:07.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.355+0000 7f088cff9700 1 --2- 172.21.15.103:0/2646523568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f087c0377b0 0x7f087c039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:07.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.355+0000 7f088cff9700 1 -- 172.21.15.103:0/2646523568 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f0884018040 con 0x7f089009c580 2023-07-02T09:04:07.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.355+0000 7f088f7fe700 1 --2- 172.21.15.103:0/2646523568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f087c0377b0 0x7f087c039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:07.358 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.356+0000 7f088f7fe700 1 --2- 172.21.15.103:0/2646523568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f087c0377b0 0x7f087c039cc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:07.361 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.360+0000 7f088cff9700 1 -- 172.21.15.103:0/2646523568 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f0884018e00 con 0x7f089009c580 2023-07-02T09:04:07.550 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.549+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0890061ea0 con 0x7f089009c580 2023-07-02T09:04:07.551 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.549+0000 7f088cff9700 1 -- 172.21.15.103:0/2646523568 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f0884010e60 con 0x7f089009c580 2023-07-02T09:04:07.551 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:07.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f087c0377b0 msgr2=0x7f087c039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:07.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 --2- 172.21.15.103:0/2646523568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f087c0377b0 0x7f087c039cc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:07.553 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f089009c580 msgr2=0x7f08901baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:07.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 --2- 172.21.15.103:0/2646523568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f089009c580 0x7f08901baa60 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:07.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 shutdown_connections 2023-07-02T09:04:07.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 --2- 172.21.15.103:0/2646523568 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f087c0377b0 0x7f087c039cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:07.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 --2- 172.21.15.103:0/2646523568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f089009c580 0x7f08901baa60 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:07.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 >> 172.21.15.103:0/2646523568 conn(0x7f08900a2080 msgr2=0x7f0890130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:07.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 shutdown_connections 2023-07-02T09:04:07.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:07.552+0000 7f0896431700 1 -- 172.21.15.103:0/2646523568 wait complete. 2023-07-02T09:04:07.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:08.567 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.701+0000 7f2ddf4ae700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.706+0000 7f2ddf4ae700 1 Processor -- start 2023-07-02T09:04:08.708 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.707+0000 7f2ddf4ae700 1 -- start start 2023-07-02T09:04:08.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.707+0000 7f2ddf4ae700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2dd8132f50 0x7f2dd8133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:08.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.707+0000 7f2ddf4ae700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd809e260 con 0x7f2dd809c580 2023-07-02T09:04:08.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.707+0000 7f2ddf4ae700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80a1640 con 0x7f2dd8132f50 2023-07-02T09:04:08.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.707+0000 7f2ddd24a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2dd8132f50 0x7f2dd8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:08.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.707+0000 7f2ddd24a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2dd8132f50 0x7f2dd8133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50466/0 (socket says 172.21.15.103:50466) 2023-07-02T09:04:08.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.707+0000 7f2ddca49700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2dd809c580 0x7f2dd809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48616/0 (socket says 172.21.15.103:48616) 2023-07-02T09:04:08.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.707+0000 7f2ddd24a700 1 -- 172.21.15.103:0/3666062866 learned_addr learned my addr 172.21.15.103:0/3666062866 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:08.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.708+0000 7f2ddd24a700 1 -- 172.21.15.103:0/3666062866 >> v1:172.21.15.103:6789/0 conn(0x7f2dd809c580 legacy=0x7f2dd809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:08.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.708+0000 7f2ddd24a700 1 -- 172.21.15.103:0/3666062866 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd8132a20 con 0x7f2dd8132f50 2023-07-02T09:04:08.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.708+0000 7f2dcffff700 1 -- 172.21.15.103:0/3666062866 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215286388 0 0) 0x7f2dd809e260 con 0x7f2dd809c580 2023-07-02T09:04:08.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.708+0000 7f2ddd24a700 1 --2- 172.21.15.103:0/3666062866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd8132f50 0x7f2dd8133360 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=371eb67ba642efaf server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:08.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.708+0000 7f2dcffff700 1 -- 172.21.15.103:0/3666062866 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2dc8009d90 con 0x7f2dd8132f50 2023-07-02T09:04:08.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.708+0000 7f2dcffff700 1 -- 172.21.15.103:0/3666062866 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2dc800e230 con 0x7f2dd8132f50 2023-07-02T09:04:08.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.708+0000 7f2dcffff700 1 -- 172.21.15.103:0/3666062866 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2dc800e4d0 con 0x7f2dd8132f50 2023-07-02T09:04:08.712 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.709+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3666062866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd8132f50 msgr2=0x7f2dd8133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:08.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.709+0000 7f2ddf4ae700 1 --2- 172.21.15.103:0/3666062866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd8132f50 0x7f2dd8133360 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:08.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.709+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3666062866 shutdown_connections 2023-07-02T09:04:08.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.709+0000 7f2ddf4ae700 1 --2- 172.21.15.103:0/3666062866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd8132f50 0x7f2dd8133360 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:08.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.709+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3666062866 >> 172.21.15.103:0/3666062866 conn(0x7f2dd80a2080 msgr2=0x7f2dd80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:08.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.709+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3666062866 shutdown_connections 2023-07-02T09:04:08.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.709+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3666062866 wait complete. 2023-07-02T09:04:08.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.709+0000 7f2ddf4ae700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:08.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.710+0000 7f2ddf4ae700 1 Processor -- start 2023-07-02T09:04:08.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.710+0000 7f2ddf4ae700 1 -- start start 2023-07-02T09:04:08.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2ddf4ae700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd809c580 0x7f2dd81baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:08.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2ddf4ae700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd81ba610 con 0x7f2dd809c580 2023-07-02T09:04:08.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2ddd24a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd809c580 0x7f2dd81baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:08.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2ddd24a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd809c580 0x7f2dd81baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50472/0 (socket says 172.21.15.103:50472) 2023-07-02T09:04:08.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2ddd24a700 1 -- 172.21.15.103:0/3307052417 learned_addr learned my addr 172.21.15.103:0/3307052417 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:08.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2ddd24a700 1 -- 172.21.15.103:0/3307052417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd806dfc0 con 0x7f2dd809c580 2023-07-02T09:04:08.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2ddd24a700 1 --2- 172.21.15.103:0/3307052417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd809c580 0x7f2dd81baa60 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:08.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2dcdffb700 1 -- 172.21.15.103:0/3307052417 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2dc8014410 con 0x7f2dd809c580 2023-07-02T09:04:08.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2dcdffb700 1 -- 172.21.15.103:0/3307052417 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2dc8014570 con 0x7f2dd809c580 2023-07-02T09:04:08.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2dcdffb700 1 -- 172.21.15.103:0/3307052417 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2dc8014810 con 0x7f2dd809c580 2023-07-02T09:04:08.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.711+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd81ba270 con 0x7f2dd809c580 2023-07-02T09:04:08.715 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.712+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd806dba0 con 0x7f2dd809c580 2023-07-02T09:04:08.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.712+0000 7f2dcdffb700 1 -- 172.21.15.103:0/3307052417 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f2dc80149c0 con 0x7f2dd809c580 2023-07-02T09:04:08.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.712+0000 7f2dcdffb700 1 --2- 172.21.15.103:0/3307052417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2dbc0377c0 0x7f2dbc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:08.716 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.713+0000 7f2dcdffb700 1 -- 172.21.15.103:0/3307052417 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f2dc80482f0 con 0x7f2dd809c580 2023-07-02T09:04:08.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.713+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd81ba460 con 0x7f2dd809c580 2023-07-02T09:04:08.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.713+0000 7f2ddca49700 1 --2- 172.21.15.103:0/3307052417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2dbc0377c0 0x7f2dbc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:08.717 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.713+0000 7f2ddca49700 1 --2- 172.21.15.103:0/3307052417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2dbc0377c0 0x7f2dbc039cd0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:08.718 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.717+0000 7f2dcdffb700 1 -- 172.21.15.103:0/3307052417 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f2dc800fb20 con 0x7f2dd809c580 2023-07-02T09:04:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.908+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2dd8061ea0 con 0x7f2dd809c580 2023-07-02T09:04:08.910 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.908+0000 7f2dcdffb700 1 -- 172.21.15.103:0/3307052417 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f2dc800f710 con 0x7f2dd809c580 2023-07-02T09:04:08.910 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2dbc0377c0 msgr2=0x7f2dbc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 --2- 172.21.15.103:0/3307052417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2dbc0377c0 0x7f2dbc039cd0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd809c580 msgr2=0x7f2dd81baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 --2- 172.21.15.103:0/3307052417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd809c580 0x7f2dd81baa60 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 shutdown_connections 2023-07-02T09:04:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 --2- 172.21.15.103:0/3307052417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f2dbc0377c0 0x7f2dbc039cd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 --2- 172.21.15.103:0/3307052417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dd809c580 0x7f2dd81baa60 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 >> 172.21.15.103:0/3307052417 conn(0x7f2dd80a2080 msgr2=0x7f2dd8130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 shutdown_connections 2023-07-02T09:04:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:08.911+0000 7f2ddf4ae700 1 -- 172.21.15.103:0/3307052417 wait complete. 2023-07-02T09:04:08.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:09.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:10.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~264s 2023-07-02T09:04:10.057 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.051+0000 7f4bd5852700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:10.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.057+0000 7f4bd5852700 1 Processor -- start 2023-07-02T09:04:10.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.057+0000 7f4bd5852700 1 -- start start 2023-07-02T09:04:10.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.057+0000 7f4bd5852700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4bd012f250 0x7f4bd012f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:10.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bd5852700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd006df60 con 0x7f4bd0098a00 2023-07-02T09:04:10.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bd5852700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd009a710 con 0x7f4bd012f250 2023-07-02T09:04:10.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bceffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4bd012f250 0x7f4bd012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:10.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bce7fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4bd0098a00 0x7f4bd0098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48630/0 (socket says 172.21.15.103:48630) 2023-07-02T09:04:10.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bce7fc700 1 -- 172.21.15.103:0/82357128 learned_addr learned my addr 172.21.15.103:0/82357128 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:10.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bceffd700 1 -- 172.21.15.103:0/82357128 >> v1:172.21.15.103:6789/0 conn(0x7f4bd0098a00 legacy=0x7f4bd0098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:10.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bceffd700 1 -- 172.21.15.103:0/82357128 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd012ed20 con 0x7f4bd012f250 2023-07-02T09:04:10.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bcdffb700 1 -- 172.21.15.103:0/82357128 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432101003 0 0) 0x7f4bd006df60 con 0x7f4bd0098a00 2023-07-02T09:04:10.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bceffd700 1 --2- 172.21.15.103:0/82357128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd012f250 0x7f4bd012f660 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459b5f4720ac2d85 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:10.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bcdffb700 1 -- 172.21.15.103:0/82357128 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4bc0009cd0 con 0x7f4bd012f250 2023-07-02T09:04:10.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bcdffb700 1 -- 172.21.15.103:0/82357128 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4bc0009e30 con 0x7f4bd012f250 2023-07-02T09:04:10.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.058+0000 7f4bcdffb700 1 -- 172.21.15.103:0/82357128 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4bc000e360 con 0x7f4bd012f250 2023-07-02T09:04:10.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.059+0000 7f4bd5852700 1 -- 172.21.15.103:0/82357128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd012f250 msgr2=0x7f4bd012f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:10.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.059+0000 7f4bd5852700 1 --2- 172.21.15.103:0/82357128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd012f250 0x7f4bd012f660 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:10.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.059+0000 7f4bd5852700 1 -- 172.21.15.103:0/82357128 shutdown_connections 2023-07-02T09:04:10.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.059+0000 7f4bd5852700 1 --2- 172.21.15.103:0/82357128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd012f250 0x7f4bd012f660 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:10.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.059+0000 7f4bd5852700 1 -- 172.21.15.103:0/82357128 >> 172.21.15.103:0/82357128 conn(0x7f4bd009d810 msgr2=0x7f4bd009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:10.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.059+0000 7f4bd5852700 1 -- 172.21.15.103:0/82357128 shutdown_connections 2023-07-02T09:04:10.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.059+0000 7f4bd5852700 1 -- 172.21.15.103:0/82357128 wait complete. 2023-07-02T09:04:10.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.059+0000 7f4bd5852700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:10.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.060+0000 7f4bd5852700 1 Processor -- start 2023-07-02T09:04:10.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.060+0000 7f4bd5852700 1 -- start start 2023-07-02T09:04:10.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.061+0000 7f4bd5852700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd0098a00 0x7f4bd0133190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:10.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.061+0000 7f4bd5852700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd0099b60 con 0x7f4bd0098a00 2023-07-02T09:04:10.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.061+0000 7f4bceffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd0098a00 0x7f4bd0133190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:10.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.061+0000 7f4bceffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd0098a00 0x7f4bd0133190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50490/0 (socket says 172.21.15.103:50490) 2023-07-02T09:04:10.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.061+0000 7f4bceffd700 1 -- 172.21.15.103:0/2218335951 learned_addr learned my addr 172.21.15.103:0/2218335951 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:10.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.061+0000 7f4bceffd700 1 -- 172.21.15.103:0/2218335951 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd0135890 con 0x7f4bd0098a00 2023-07-02T09:04:10.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.061+0000 7f4bceffd700 1 --2- 172.21.15.103:0/2218335951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd0098a00 0x7f4bd0133190 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:10.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.062+0000 7f4bb7fff700 1 -- 172.21.15.103:0/2218335951 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4bc0014540 con 0x7f4bd0098a00 2023-07-02T09:04:10.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.062+0000 7f4bb7fff700 1 -- 172.21.15.103:0/2218335951 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4bc00146a0 con 0x7f4bd0098a00 2023-07-02T09:04:10.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.062+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd012c6c0 con 0x7f4bd0098a00 2023-07-02T09:04:10.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.062+0000 7f4bb7fff700 1 -- 172.21.15.103:0/2218335951 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4bc0014940 con 0x7f4bd0098a00 2023-07-02T09:04:10.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.062+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd0119570 con 0x7f4bd0098a00 2023-07-02T09:04:10.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.062+0000 7f4bb7fff700 1 -- 172.21.15.103:0/2218335951 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f4bc0014af0 con 0x7f4bd0098a00 2023-07-02T09:04:10.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.063+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd0061ea0 con 0x7f4bd0098a00 2023-07-02T09:04:10.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.063+0000 7f4bb7fff700 1 --2- 172.21.15.103:0/2218335951 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4bbc0377b0 0x7f4bbc039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:10.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.063+0000 7f4bb7fff700 1 -- 172.21.15.103:0/2218335951 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f4bc0048270 con 0x7f4bd0098a00 2023-07-02T09:04:10.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.063+0000 7f4bce7fc700 1 --2- 172.21.15.103:0/2218335951 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4bbc0377b0 0x7f4bbc039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:10.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.063+0000 7f4bce7fc700 1 --2- 172.21.15.103:0/2218335951 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4bbc0377b0 0x7f4bbc039cc0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:10.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.067+0000 7f4bb7fff700 1 -- 172.21.15.103:0/2218335951 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f4bc000fe40 con 0x7f4bd0098a00 2023-07-02T09:04:10.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.256+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4bd0073ca0 con 0x7f4bd0098a00 2023-07-02T09:04:10.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.257+0000 7f4bb7fff700 1 -- 172.21.15.103:0/2218335951 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f4bc000fa50 con 0x7f4bd0098a00 2023-07-02T09:04:10.259 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:10.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4bbc0377b0 msgr2=0x7f4bbc039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:10.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 --2- 172.21.15.103:0/2218335951 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4bbc0377b0 0x7f4bbc039cc0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:10.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd0098a00 msgr2=0x7f4bd0133190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:10.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 --2- 172.21.15.103:0/2218335951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd0098a00 0x7f4bd0133190 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:10.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 shutdown_connections 2023-07-02T09:04:10.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 --2- 172.21.15.103:0/2218335951 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f4bbc0377b0 0x7f4bbc039cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:10.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 --2- 172.21.15.103:0/2218335951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd0098a00 0x7f4bd0133190 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:10.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 >> 172.21.15.103:0/2218335951 conn(0x7f4bd009d810 msgr2=0x7f4bd009f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:10.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 shutdown_connections 2023-07-02T09:04:10.261 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:10.259+0000 7f4bd5852700 1 -- 172.21.15.103:0/2218335951 wait complete. 2023-07-02T09:04:10.273 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:11.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:11.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.414+0000 7fa4fcf49700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:11.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.419+0000 7fa4fcf49700 1 Processor -- start 2023-07-02T09:04:11.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.419+0000 7fa4fcf49700 1 -- start start 2023-07-02T09:04:11.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4fcf49700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa4f8132f50 0x7fa4f8133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:11.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4fcf49700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f809e260 con 0x7fa4f809c580 2023-07-02T09:04:11.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4fcf49700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f80a1640 con 0x7fa4f8132f50 2023-07-02T09:04:11.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4f659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa4f8132f50 0x7fa4f8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:11.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4f5d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa4f809c580 0x7fa4f809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48644/0 (socket says 172.21.15.103:48644) 2023-07-02T09:04:11.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4f5d9b700 1 -- 172.21.15.103:0/3264958496 learned_addr learned my addr 172.21.15.103:0/3264958496 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:11.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4f659c700 1 -- 172.21.15.103:0/3264958496 >> v1:172.21.15.103:6789/0 conn(0x7fa4f809c580 legacy=0x7fa4f809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:11.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4f659c700 1 -- 172.21.15.103:0/3264958496 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8132a20 con 0x7fa4f8132f50 2023-07-02T09:04:11.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4f559a700 1 -- 172.21.15.103:0/3264958496 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849183426 0 0) 0x7fa4f809e260 con 0x7fa4f809c580 2023-07-02T09:04:11.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4f659c700 1 --2- 172.21.15.103:0/3264958496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f8132f50 0x7fa4f8133360 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da519e82dfcedc6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:11.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.420+0000 7fa4f559a700 1 -- 172.21.15.103:0/3264958496 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa4ec009cd0 con 0x7fa4f8132f50 2023-07-02T09:04:11.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4f559a700 1 -- 172.21.15.103:0/3264958496 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa4ec009e30 con 0x7fa4f8132f50 2023-07-02T09:04:11.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4f559a700 1 -- 172.21.15.103:0/3264958496 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa4ec00e360 con 0x7fa4f8132f50 2023-07-02T09:04:11.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4fcf49700 1 -- 172.21.15.103:0/3264958496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f8132f50 msgr2=0x7fa4f8133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:11.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4fcf49700 1 --2- 172.21.15.103:0/3264958496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f8132f50 0x7fa4f8133360 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:11.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4fcf49700 1 -- 172.21.15.103:0/3264958496 shutdown_connections 2023-07-02T09:04:11.424 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4fcf49700 1 --2- 172.21.15.103:0/3264958496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f8132f50 0x7fa4f8133360 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:11.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4fcf49700 1 -- 172.21.15.103:0/3264958496 >> 172.21.15.103:0/3264958496 conn(0x7fa4f80a2080 msgr2=0x7fa4f80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:11.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4fcf49700 1 -- 172.21.15.103:0/3264958496 shutdown_connections 2023-07-02T09:04:11.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.421+0000 7fa4fcf49700 1 -- 172.21.15.103:0/3264958496 wait complete. 2023-07-02T09:04:11.425 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.422+0000 7fa4fcf49700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:11.426 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.423+0000 7fa4fcf49700 1 Processor -- start 2023-07-02T09:04:11.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.423+0000 7fa4fcf49700 1 -- start start 2023-07-02T09:04:11.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.423+0000 7fa4fcf49700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f809c580 0x7fa4f812beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:11.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.423+0000 7fa4fcf49700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f8133a00 con 0x7fa4f809c580 2023-07-02T09:04:11.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.423+0000 7fa4f659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f809c580 0x7fa4f812beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:11.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4f659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f809c580 0x7fa4f812beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50504/0 (socket says 172.21.15.103:50504) 2023-07-02T09:04:11.427 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4f659c700 1 -- 172.21.15.103:0/2649578106 learned_addr learned my addr 172.21.15.103:0/2649578106 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:11.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4f659c700 1 -- 172.21.15.103:0/2649578106 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f806dc00 con 0x7fa4f809c580 2023-07-02T09:04:11.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4f659c700 1 --2- 172.21.15.103:0/2649578106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f809c580 0x7fa4f812beb0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:11.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4e6ffd700 1 -- 172.21.15.103:0/2649578106 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa4ec013530 con 0x7fa4f809c580 2023-07-02T09:04:11.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4e6ffd700 1 -- 172.21.15.103:0/2649578106 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa4ec013690 con 0x7fa4f809c580 2023-07-02T09:04:11.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4e6ffd700 1 -- 172.21.15.103:0/2649578106 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa4ec013930 con 0x7fa4f809c580 2023-07-02T09:04:11.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f806e920 con 0x7fa4f809c580 2023-07-02T09:04:11.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.424+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f8071f80 con 0x7fa4f809c580 2023-07-02T09:04:11.428 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.425+0000 7fa4e6ffd700 1 -- 172.21.15.103:0/2649578106 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa4ec013ae0 con 0x7fa4f809c580 2023-07-02T09:04:11.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.425+0000 7fa4e6ffd700 1 --2- 172.21.15.103:0/2649578106 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa4d8037800 0x7fa4d8039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:11.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.425+0000 7fa4e6ffd700 1 -- 172.21.15.103:0/2649578106 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa4ec018040 con 0x7fa4f809c580 2023-07-02T09:04:11.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.425+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f8061ea0 con 0x7fa4f809c580 2023-07-02T09:04:11.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.425+0000 7fa4f5d9b700 1 --2- 172.21.15.103:0/2649578106 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa4d8037800 0x7fa4d8039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:11.429 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.426+0000 7fa4f5d9b700 1 --2- 172.21.15.103:0/2649578106 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa4d8037800 0x7fa4d8039d10 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:11.431 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.430+0000 7fa4e6ffd700 1 -- 172.21.15.103:0/2649578106 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa4ec018e00 con 0x7fa4f809c580 2023-07-02T09:04:11.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.618+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa4f80725b0 con 0x7fa4f809c580 2023-07-02T09:04:11.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.619+0000 7fa4e6ffd700 1 -- 172.21.15.103:0/2649578106 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa4ec010e60 con 0x7fa4f809c580 2023-07-02T09:04:11.620 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:11.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa4d8037800 msgr2=0x7fa4d8039d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:11.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 --2- 172.21.15.103:0/2649578106 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa4d8037800 0x7fa4d8039d10 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:11.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f809c580 msgr2=0x7fa4f812beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:11.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 --2- 172.21.15.103:0/2649578106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f809c580 0x7fa4f812beb0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:11.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 shutdown_connections 2023-07-02T09:04:11.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 --2- 172.21.15.103:0/2649578106 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa4d8037800 0x7fa4d8039d10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:11.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 --2- 172.21.15.103:0/2649578106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4f809c580 0x7fa4f812beb0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:11.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 >> 172.21.15.103:0/2649578106 conn(0x7fa4f80a2080 msgr2=0x7fa4f81ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:11.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 shutdown_connections 2023-07-02T09:04:11.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:11.621+0000 7fa4fcf49700 1 -- 172.21.15.103:0/2649578106 wait complete. 2023-07-02T09:04:11.635 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:12.636 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.774+0000 7f13134a7700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:12.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.779+0000 7f13134a7700 1 Processor -- start 2023-07-02T09:04:12.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.779+0000 7f13134a7700 1 -- start start 2023-07-02T09:04:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f13134a7700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f130c132f50 0x7f130c133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f13134a7700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c09e260 con 0x7f130c09c580 2023-07-02T09:04:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f13134a7700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c0a1640 con 0x7f130c132f50 2023-07-02T09:04:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f1311243700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f130c132f50 0x7f130c133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f1310a42700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f130c09c580 0x7f130c09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:48660/0 (socket says 172.21.15.103:48660) 2023-07-02T09:04:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f1310a42700 1 -- 172.21.15.103:0/141715947 learned_addr learned my addr 172.21.15.103:0/141715947 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f1311243700 1 -- 172.21.15.103:0/141715947 >> v1:172.21.15.103:6789/0 conn(0x7f130c09c580 legacy=0x7f130c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f1311243700 1 -- 172.21.15.103:0/141715947 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c132a20 con 0x7f130c132f50 2023-07-02T09:04:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.780+0000 7f1303fff700 1 -- 172.21.15.103:0/141715947 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225306488 0 0) 0x7f130c09e260 con 0x7f130c09c580 2023-07-02T09:04:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f1311243700 1 --2- 172.21.15.103:0/141715947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c132f50 0x7f130c133360 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75288a449c7ac5b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f1303fff700 1 -- 172.21.15.103:0/141715947 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f12fc009cd0 con 0x7f130c132f50 2023-07-02T09:04:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f1303fff700 1 -- 172.21.15.103:0/141715947 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f12fc009e30 con 0x7f130c132f50 2023-07-02T09:04:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f1303fff700 1 -- 172.21.15.103:0/141715947 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f12fc00e360 con 0x7f130c132f50 2023-07-02T09:04:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f13134a7700 1 -- 172.21.15.103:0/141715947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c132f50 msgr2=0x7f130c133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f13134a7700 1 --2- 172.21.15.103:0/141715947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c132f50 0x7f130c133360 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f13134a7700 1 -- 172.21.15.103:0/141715947 shutdown_connections 2023-07-02T09:04:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f13134a7700 1 --2- 172.21.15.103:0/141715947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c132f50 0x7f130c133360 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f13134a7700 1 -- 172.21.15.103:0/141715947 >> 172.21.15.103:0/141715947 conn(0x7f130c0a2080 msgr2=0x7f130c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f13134a7700 1 -- 172.21.15.103:0/141715947 shutdown_connections 2023-07-02T09:04:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.781+0000 7f13134a7700 1 -- 172.21.15.103:0/141715947 wait complete. 2023-07-02T09:04:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.782+0000 7f13134a7700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.783+0000 7f13134a7700 1 Processor -- start 2023-07-02T09:04:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.783+0000 7f13134a7700 1 -- start start 2023-07-02T09:04:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.783+0000 7f13134a7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c09c580 0x7f130c12beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.783+0000 7f13134a7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c133a00 con 0x7f130c09c580 2023-07-02T09:04:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.783+0000 7f1311243700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c09c580 0x7f130c12beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.783+0000 7f1311243700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c09c580 0x7f130c12beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50534/0 (socket says 172.21.15.103:50534) 2023-07-02T09:04:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.783+0000 7f1311243700 1 -- 172.21.15.103:0/557223302 learned_addr learned my addr 172.21.15.103:0/557223302 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.784+0000 7f1311243700 1 -- 172.21.15.103:0/557223302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c06dc00 con 0x7f130c09c580 2023-07-02T09:04:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.784+0000 7f1311243700 1 --2- 172.21.15.103:0/557223302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c09c580 0x7f130c12beb0 crc :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.784+0000 7f1301ffb700 1 -- 172.21.15.103:0/557223302 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f12fc014540 con 0x7f130c09c580 2023-07-02T09:04:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.784+0000 7f1301ffb700 1 -- 172.21.15.103:0/557223302 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f12fc0146a0 con 0x7f130c09c580 2023-07-02T09:04:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.784+0000 7f1301ffb700 1 -- 172.21.15.103:0/557223302 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f12fc014940 con 0x7f130c09c580 2023-07-02T09:04:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.784+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c06e920 con 0x7f130c09c580 2023-07-02T09:04:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.784+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c071f80 con 0x7f130c09c580 2023-07-02T09:04:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.785+0000 7f1301ffb700 1 -- 172.21.15.103:0/557223302 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f12fc014af0 con 0x7f130c09c580 2023-07-02T09:04:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.785+0000 7f1301ffb700 1 --2- 172.21.15.103:0/557223302 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f12f00377c0 0x7f12f0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.785+0000 7f1301ffb700 1 -- 172.21.15.103:0/557223302 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f12fc048340 con 0x7f130c09c580 2023-07-02T09:04:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.785+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c061ea0 con 0x7f130c09c580 2023-07-02T09:04:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.785+0000 7f1310a42700 1 --2- 172.21.15.103:0/557223302 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f12f00377c0 0x7f12f0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.786+0000 7f1310a42700 1 --2- 172.21.15.103:0/557223302 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f12f00377c0 0x7f12f0039cd0 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.790+0000 7f1301ffb700 1 -- 172.21.15.103:0/557223302 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f12fc014df0 con 0x7f130c09c580 2023-07-02T09:04:12.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.985+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f130c0725b0 con 0x7f130c09c580 2023-07-02T09:04:12.987 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.986+0000 7f1301ffb700 1 -- 172.21.15.103:0/557223302 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f12fc011320 con 0x7f130c09c580 2023-07-02T09:04:12.987 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f12f00377c0 msgr2=0x7f12f0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 --2- 172.21.15.103:0/557223302 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f12f00377c0 0x7f12f0039cd0 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c09c580 msgr2=0x7f130c12beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 --2- 172.21.15.103:0/557223302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c09c580 0x7f130c12beb0 crc :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 shutdown_connections 2023-07-02T09:04:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 --2- 172.21.15.103:0/557223302 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f12f00377c0 0x7f12f0039cd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 --2- 172.21.15.103:0/557223302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f130c09c580 0x7f130c12beb0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 >> 172.21.15.103:0/557223302 conn(0x7f130c0a2080 msgr2=0x7f130c1ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 shutdown_connections 2023-07-02T09:04:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:12.988+0000 7f13134a7700 1 -- 172.21.15.103:0/557223302 wait complete. 2023-07-02T09:04:13.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:14.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:14.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.143+0000 7f36d80ec700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:14.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.148+0000 7f36d80ec700 1 Processor -- start 2023-07-02T09:04:14.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.148+0000 7f36d80ec700 1 -- start start 2023-07-02T09:04:14.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.148+0000 7f36d80ec700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36d012f250 0x7f36d012f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:14.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.148+0000 7f36d80ec700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d006df60 con 0x7f36d0098a00 2023-07-02T09:04:14.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d80ec700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d009a710 con 0x7f36d012f250 2023-07-02T09:04:14.150 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d5e88700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36d012f250 0x7f36d012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d5687700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f36d0098a00 0x7f36d0098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43698/0 (socket says 172.21.15.103:43698) 2023-07-02T09:04:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d5687700 1 -- 172.21.15.103:0/688481384 learned_addr learned my addr 172.21.15.103:0/688481384 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d5e88700 1 -- 172.21.15.103:0/688481384 >> v1:172.21.15.103:6789/0 conn(0x7f36d0098a00 legacy=0x7f36d0098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d5e88700 1 -- 172.21.15.103:0/688481384 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d012ed20 con 0x7f36d012f250 2023-07-02T09:04:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d5e88700 1 --2- 172.21.15.103:0/688481384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d012f250 0x7f36d012f660 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e6a58e43669292 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:14.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d4e86700 1 -- 172.21.15.103:0/688481384 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36c0009cd0 con 0x7f36d012f250 2023-07-02T09:04:14.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d4e86700 1 -- 172.21.15.103:0/688481384 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f36c0009e30 con 0x7f36d012f250 2023-07-02T09:04:14.152 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.149+0000 7f36d4e86700 1 -- 172.21.15.103:0/688481384 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36c000e360 con 0x7f36d012f250 2023-07-02T09:04:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.150+0000 7f36d80ec700 1 -- 172.21.15.103:0/688481384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d012f250 msgr2=0x7f36d012f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.150+0000 7f36d80ec700 1 --2- 172.21.15.103:0/688481384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d012f250 0x7f36d012f660 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.150+0000 7f36d80ec700 1 -- 172.21.15.103:0/688481384 shutdown_connections 2023-07-02T09:04:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.150+0000 7f36d80ec700 1 --2- 172.21.15.103:0/688481384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d012f250 0x7f36d012f660 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.150+0000 7f36d80ec700 1 -- 172.21.15.103:0/688481384 >> 172.21.15.103:0/688481384 conn(0x7f36d009d810 msgr2=0x7f36d009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.150+0000 7f36d80ec700 1 -- 172.21.15.103:0/688481384 shutdown_connections 2023-07-02T09:04:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.150+0000 7f36d80ec700 1 -- 172.21.15.103:0/688481384 wait complete. 2023-07-02T09:04:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.150+0000 7f36d80ec700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.151+0000 7f36d80ec700 1 Processor -- start 2023-07-02T09:04:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.151+0000 7f36d80ec700 1 -- start start 2023-07-02T09:04:14.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d80ec700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d0098a00 0x7f36d01c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:14.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d80ec700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d011c9e0 con 0x7f36d0098a00 2023-07-02T09:04:14.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d5e88700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d0098a00 0x7f36d01c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:14.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d5e88700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d0098a00 0x7f36d01c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38168/0 (socket says 172.21.15.103:38168) 2023-07-02T09:04:14.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d5e88700 1 -- 172.21.15.103:0/1392691601 learned_addr learned my addr 172.21.15.103:0/1392691601 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:14.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d5e88700 1 -- 172.21.15.103:0/1392691601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d011aaa0 con 0x7f36d0098a00 2023-07-02T09:04:14.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d5e88700 1 --2- 172.21.15.103:0/1392691601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d0098a00 0x7f36d01c3e20 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:14.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36c67fc700 1 -- 172.21.15.103:0/1392691601 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36c0017040 con 0x7f36d0098a00 2023-07-02T09:04:14.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36c67fc700 1 -- 172.21.15.103:0/1392691601 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f36c000e690 con 0x7f36d0098a00 2023-07-02T09:04:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d0119810 con 0x7f36d0098a00 2023-07-02T09:04:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36c67fc700 1 -- 172.21.15.103:0/1392691601 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36c000e930 con 0x7f36d0098a00 2023-07-02T09:04:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.152+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d0129120 con 0x7f36d0098a00 2023-07-02T09:04:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.153+0000 7f36c67fc700 1 -- 172.21.15.103:0/1392691601 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f36c001a040 con 0x7f36d0098a00 2023-07-02T09:04:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.153+0000 7f36c67fc700 1 --2- 172.21.15.103:0/1392691601 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36b40377c0 0x7f36b4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.153+0000 7f36c67fc700 1 -- 172.21.15.103:0/1392691601 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f36c0049e80 con 0x7f36d0098a00 2023-07-02T09:04:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.154+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d0128f70 con 0x7f36d0098a00 2023-07-02T09:04:14.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.154+0000 7f36d5687700 1 --2- 172.21.15.103:0/1392691601 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36b40377c0 0x7f36b4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:14.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.154+0000 7f36d5687700 1 --2- 172.21.15.103:0/1392691601 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36b40377c0 0x7f36b4039cd0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:14.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.158+0000 7f36c67fc700 1 -- 172.21.15.103:0/1392691601 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f36c00175b0 con 0x7f36d0098a00 2023-07-02T09:04:14.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.348+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f36d009c800 con 0x7f36d0098a00 2023-07-02T09:04:14.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.348+0000 7f36c67fc700 1 -- 172.21.15.103:0/1392691601 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f36c0007000 con 0x7f36d0098a00 2023-07-02T09:04:14.350 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:14.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36b40377c0 msgr2=0x7f36b4039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:14.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 --2- 172.21.15.103:0/1392691601 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36b40377c0 0x7f36b4039cd0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:14.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d0098a00 msgr2=0x7f36d01c3e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:14.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 --2- 172.21.15.103:0/1392691601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d0098a00 0x7f36d01c3e20 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:14.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 shutdown_connections 2023-07-02T09:04:14.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 --2- 172.21.15.103:0/1392691601 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36b40377c0 0x7f36b4039cd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:14.352 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 --2- 172.21.15.103:0/1392691601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36d0098a00 0x7f36d01c3e20 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:14.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 >> 172.21.15.103:0/1392691601 conn(0x7f36d009d810 msgr2=0x7f36d009eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:14.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 shutdown_connections 2023-07-02T09:04:14.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:14.350+0000 7f36d80ec700 1 -- 172.21.15.103:0/1392691601 wait complete. 2023-07-02T09:04:14.364 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:15.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:15.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.503+0000 7fefd3187700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:15.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.508+0000 7fefd3187700 1 Processor -- start 2023-07-02T09:04:15.510 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.508+0000 7fefd3187700 1 -- start start 2023-07-02T09:04:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.509+0000 7fefd3187700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fefcc132f50 0x7fefcc133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.509+0000 7fefd3187700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefcc09e260 con 0x7fefcc09c580 2023-07-02T09:04:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.509+0000 7fefd3187700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc0a1640 con 0x7fefcc132f50 2023-07-02T09:04:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.509+0000 7fefd0f23700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fefcc132f50 0x7fefcc133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.509+0000 7fefd0f23700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fefcc132f50 0x7fefcc133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38176/0 (socket says 172.21.15.103:38176) 2023-07-02T09:04:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.509+0000 7fefcbfff700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fefcc09c580 0x7fefcc09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43708/0 (socket says 172.21.15.103:43708) 2023-07-02T09:04:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.509+0000 7fefd0f23700 1 -- 172.21.15.103:0/591870507 learned_addr learned my addr 172.21.15.103:0/591870507 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:15.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.510+0000 7fefd0f23700 1 -- 172.21.15.103:0/591870507 >> v1:172.21.15.103:6789/0 conn(0x7fefcc09c580 legacy=0x7fefcc09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:15.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.510+0000 7fefcb7fe700 1 -- 172.21.15.103:0/591870507 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406196794 0 0) 0x7fefcc09e260 con 0x7fefcc09c580 2023-07-02T09:04:15.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.510+0000 7fefd0f23700 1 -- 172.21.15.103:0/591870507 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc132a20 con 0x7fefcc132f50 2023-07-02T09:04:15.512 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.510+0000 7fefd0f23700 1 --2- 172.21.15.103:0/591870507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc132f50 0x7fefcc133360 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7894debd0904dd90 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:15.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.510+0000 7fefcb7fe700 1 -- 172.21.15.103:0/591870507 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fefc0009d90 con 0x7fefcc132f50 2023-07-02T09:04:15.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.510+0000 7fefcb7fe700 1 -- 172.21.15.103:0/591870507 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fefc000e230 con 0x7fefcc132f50 2023-07-02T09:04:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.510+0000 7fefcb7fe700 1 -- 172.21.15.103:0/591870507 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fefc000e4d0 con 0x7fefcc132f50 2023-07-02T09:04:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.511+0000 7fefd3187700 1 -- 172.21.15.103:0/591870507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc132f50 msgr2=0x7fefcc133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.511+0000 7fefd3187700 1 --2- 172.21.15.103:0/591870507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc132f50 0x7fefcc133360 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.511+0000 7fefd3187700 1 -- 172.21.15.103:0/591870507 shutdown_connections 2023-07-02T09:04:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.511+0000 7fefd3187700 1 --2- 172.21.15.103:0/591870507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc132f50 0x7fefcc133360 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.511+0000 7fefd3187700 1 -- 172.21.15.103:0/591870507 >> 172.21.15.103:0/591870507 conn(0x7fefcc0a2080 msgr2=0x7fefcc0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.511+0000 7fefd3187700 1 -- 172.21.15.103:0/591870507 shutdown_connections 2023-07-02T09:04:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.511+0000 7fefd3187700 1 -- 172.21.15.103:0/591870507 wait complete. 2023-07-02T09:04:15.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.511+0000 7fefd3187700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:15.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.512+0000 7fefd3187700 1 Processor -- start 2023-07-02T09:04:15.515 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.512+0000 7fefd3187700 1 -- start start 2023-07-02T09:04:15.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd3187700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc09c580 0x7fefcc12beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd3187700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc133a00 con 0x7fefcc09c580 2023-07-02T09:04:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd0f23700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc09c580 0x7fefcc12beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd0f23700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc09c580 0x7fefcc12beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38186/0 (socket says 172.21.15.103:38186) 2023-07-02T09:04:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd0f23700 1 -- 172.21.15.103:0/3877051608 learned_addr learned my addr 172.21.15.103:0/3877051608 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd0f23700 1 -- 172.21.15.103:0/3877051608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc06dc00 con 0x7fefcc09c580 2023-07-02T09:04:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd0f23700 1 --2- 172.21.15.103:0/3877051608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc09c580 0x7fefcc12beb0 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefc97fa700 1 -- 172.21.15.103:0/3877051608 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fefc0014540 con 0x7fefcc09c580 2023-07-02T09:04:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefc97fa700 1 -- 172.21.15.103:0/3877051608 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fefc00146a0 con 0x7fefcc09c580 2023-07-02T09:04:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc06e920 con 0x7fefcc09c580 2023-07-02T09:04:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefc97fa700 1 -- 172.21.15.103:0/3877051608 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fefc0014940 con 0x7fefcc09c580 2023-07-02T09:04:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.513+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc071f80 con 0x7fefcc09c580 2023-07-02T09:04:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.514+0000 7fefc97fa700 1 -- 172.21.15.103:0/3877051608 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fefc0014af0 con 0x7fefcc09c580 2023-07-02T09:04:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.514+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc061ea0 con 0x7fefcc09c580 2023-07-02T09:04:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.514+0000 7fefc97fa700 1 --2- 172.21.15.103:0/3877051608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fefb00377c0 0x7fefb0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:15.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.514+0000 7fefc97fa700 1 -- 172.21.15.103:0/3877051608 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fefc00483f0 con 0x7fefcc09c580 2023-07-02T09:04:15.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.515+0000 7fefcbfff700 1 --2- 172.21.15.103:0/3877051608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fefb00377c0 0x7fefb0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:15.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.515+0000 7fefcbfff700 1 --2- 172.21.15.103:0/3877051608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fefb00377c0 0x7fefb0039cd0 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:15.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.519+0000 7fefc97fa700 1 -- 172.21.15.103:0/3877051608 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fefc000fc80 con 0x7fefcc09c580 2023-07-02T09:04:15.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2023-07-02T09:04:15.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.709+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fefcc0725b0 con 0x7fefcc09c580 2023-07-02T09:04:15.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.709+0000 7fefc97fa700 1 -- 172.21.15.103:0/3877051608 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fefc000fa50 con 0x7fefcc09c580 2023-07-02T09:04:15.711 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:15.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.711+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fefb00377c0 msgr2=0x7fefb0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:15.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.711+0000 7fefd3187700 1 --2- 172.21.15.103:0/3877051608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fefb00377c0 0x7fefb0039cd0 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:15.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.711+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc09c580 msgr2=0x7fefcc12beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:15.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.711+0000 7fefd3187700 1 --2- 172.21.15.103:0/3877051608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc09c580 0x7fefcc12beb0 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:15.713 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.712+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 shutdown_connections 2023-07-02T09:04:15.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.712+0000 7fefd3187700 1 --2- 172.21.15.103:0/3877051608 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fefb00377c0 0x7fefb0039cd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:15.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.712+0000 7fefd3187700 1 --2- 172.21.15.103:0/3877051608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefcc09c580 0x7fefcc12beb0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:15.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.712+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 >> 172.21.15.103:0/3877051608 conn(0x7fefcc0a2080 msgr2=0x7fefcc1ba9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:15.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.712+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 shutdown_connections 2023-07-02T09:04:15.714 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:15.712+0000 7fefd3187700 1 -- 172.21.15.103:0/3877051608 wait complete. 2023-07-02T09:04:15.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:16.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:16.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.864+0000 7fa1a6eb0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:16.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.869+0000 7fa1a6eb0700 1 Processor -- start 2023-07-02T09:04:16.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.869+0000 7fa1a6eb0700 1 -- start start 2023-07-02T09:04:16.871 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa1a6eb0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1a009a220 0x7fa1a01348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:16.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa1a6eb0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a009cea0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa1a6eb0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a006d490 con 0x7fa1a009a220 2023-07-02T09:04:16.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1a009a220 0x7fa1a01348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:16.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19ffff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1a009a220 0x7fa1a01348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38202/0 (socket says 172.21.15.103:38202) 2023-07-02T09:04:16.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19ffff700 1 -- 172.21.15.103:0/1173843143 learned_addr learned my addr 172.21.15.103:0/1173843143 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:16.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa1a4c4c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa1a006d6e0 0x7fa1a0099b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43718/0 (socket says 172.21.15.103:43718) 2023-07-02T09:04:16.872 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19ffff700 1 -- 172.21.15.103:0/1173843143 >> v1:172.21.15.103:6789/0 conn(0x7fa1a006d6e0 legacy=0x7fa1a0099b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:16.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19ffff700 1 -- 172.21.15.103:0/1173843143 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a006dc80 con 0x7fa1a009a220 2023-07-02T09:04:16.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19f7fe700 1 -- 172.21.15.103:0/1173843143 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778499724 0 0) 0x7fa1a009cea0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19ffff700 1 --2- 172.21.15.103:0/1173843143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009a220 0x7fa1a01348e0 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2fcbd1b9203201 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:16.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19f7fe700 1 -- 172.21.15.103:0/1173843143 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa18c009d90 con 0x7fa1a009a220 2023-07-02T09:04:16.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19f7fe700 1 -- 172.21.15.103:0/1173843143 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa18c00e230 con 0x7fa1a009a220 2023-07-02T09:04:16.873 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.870+0000 7fa19f7fe700 1 -- 172.21.15.103:0/1173843143 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa18c00e4d0 con 0x7fa1a009a220 2023-07-02T09:04:16.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.871+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/1173843143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009a220 msgr2=0x7fa1a01348e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:16.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.871+0000 7fa1a6eb0700 1 --2- 172.21.15.103:0/1173843143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009a220 0x7fa1a01348e0 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:16.875 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.871+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/1173843143 shutdown_connections 2023-07-02T09:04:16.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.871+0000 7fa1a6eb0700 1 --2- 172.21.15.103:0/1173843143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a009a220 0x7fa1a01348e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:16.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.871+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/1173843143 >> 172.21.15.103:0/1173843143 conn(0x7fa1a009d810 msgr2=0x7fa1a009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:16.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.871+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/1173843143 shutdown_connections 2023-07-02T09:04:16.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.871+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/1173843143 wait complete. 2023-07-02T09:04:16.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.871+0000 7fa1a6eb0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:16.876 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.872+0000 7fa1a6eb0700 1 Processor -- start 2023-07-02T09:04:16.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.872+0000 7fa1a6eb0700 1 -- start start 2023-07-02T09:04:16.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.873+0000 7fa1a6eb0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a006d6e0 0x7fa1a011db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:16.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.873+0000 7fa1a6eb0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a011f0e0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.873+0000 7fa1a4c4c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a006d6e0 0x7fa1a011db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:16.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.873+0000 7fa1a4c4c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a006d6e0 0x7fa1a011db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38216/0 (socket says 172.21.15.103:38216) 2023-07-02T09:04:16.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.873+0000 7fa1a4c4c700 1 -- 172.21.15.103:0/13162994 learned_addr learned my addr 172.21.15.103:0/13162994 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:16.877 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.874+0000 7fa1a4c4c700 1 -- 172.21.15.103:0/13162994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a01244e0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.874+0000 7fa1a4c4c700 1 --2- 172.21.15.103:0/13162994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a006d6e0 0x7fa1a011db90 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:16.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.874+0000 7fa19d7fa700 1 -- 172.21.15.103:0/13162994 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa194004b80 con 0x7fa1a006d6e0 2023-07-02T09:04:16.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.874+0000 7fa19d7fa700 1 -- 172.21.15.103:0/13162994 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa194004ce0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.874+0000 7fa19d7fa700 1 -- 172.21.15.103:0/13162994 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa19400c550 con 0x7fa1a006d6e0 2023-07-02T09:04:16.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.874+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a0124fa0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.878 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.874+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a012f2d0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.875+0000 7fa19d7fa700 1 -- 172.21.15.103:0/13162994 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa19400c6b0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.875+0000 7fa19d7fa700 1 --2- 172.21.15.103:0/13162994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1840377c0 0x7fa184039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:16.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.875+0000 7fa19d7fa700 1 -- 172.21.15.103:0/13162994 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa1940469d0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.879 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.875+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a0061ea0 con 0x7fa1a006d6e0 2023-07-02T09:04:16.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.875+0000 7fa19ffff700 1 --2- 172.21.15.103:0/13162994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1840377c0 0x7fa184039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:16.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.876+0000 7fa19ffff700 1 --2- 172.21.15.103:0/13162994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1840377c0 0x7fa184039cd0 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:16.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:16.880+0000 7fa19d7fa700 1 -- 172.21.15.103:0/13162994 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa194009c80 con 0x7fa1a006d6e0 2023-07-02T09:04:17.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.070+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa1a0070fe0 con 0x7fa1a006d6e0 2023-07-02T09:04:17.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.071+0000 7fa19d7fa700 1 -- 172.21.15.103:0/13162994 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa1940098a0 con 0x7fa1a006d6e0 2023-07-02T09:04:17.072 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:17.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1840377c0 msgr2=0x7fa184039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:17.074 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 --2- 172.21.15.103:0/13162994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1840377c0 0x7fa184039cd0 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:17.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a006d6e0 msgr2=0x7fa1a011db90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:17.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 --2- 172.21.15.103:0/13162994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a006d6e0 0x7fa1a011db90 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:17.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 shutdown_connections 2023-07-02T09:04:17.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 --2- 172.21.15.103:0/13162994 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa1840377c0 0x7fa184039cd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:17.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 --2- 172.21.15.103:0/13162994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1a006d6e0 0x7fa1a011db90 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:17.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 >> 172.21.15.103:0/13162994 conn(0x7fa1a009d810 msgr2=0x7fa1a0117360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:17.075 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 shutdown_connections 2023-07-02T09:04:17.076 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:17.073+0000 7fa1a6eb0700 1 -- 172.21.15.103:0/13162994 wait complete. 2023-07-02T09:04:17.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:18.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.220+0000 7f7b51543700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.225+0000 7f7b51543700 1 Processor -- start 2023-07-02T09:04:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.225+0000 7f7b51543700 1 -- start start 2023-07-02T09:04:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b51543700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b4c133360 0x7f7b4c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b51543700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c09e260 con 0x7f7b4c132f50 2023-07-02T09:04:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b51543700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c0a1640 con 0x7f7b4c133360 2023-07-02T09:04:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b4a7fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b4c133360 0x7f7b4c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b4a7fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b4c133360 0x7f7b4c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38220/0 (socket says 172.21.15.103:38220) 2023-07-02T09:04:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b4affd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7b4c132f50 0x7f7b4c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43724/0 (socket says 172.21.15.103:43724) 2023-07-02T09:04:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b4a7fc700 1 -- 172.21.15.103:0/2404284083 learned_addr learned my addr 172.21.15.103:0/2404284083 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b4a7fc700 1 -- 172.21.15.103:0/2404284083 >> v1:172.21.15.103:6789/0 conn(0x7f7b4c132f50 legacy=0x7f7b4c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b4a7fc700 1 -- 172.21.15.103:0/2404284083 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c132a20 con 0x7f7b4c133360 2023-07-02T09:04:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b49ffb700 1 -- 172.21.15.103:0/2404284083 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659011583 0 0) 0x7f7b4c09e260 con 0x7f7b4c132f50 2023-07-02T09:04:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b4a7fc700 1 --2- 172.21.15.103:0/2404284083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c133360 0x7f7b4c09cc90 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1892a1d0240cd9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.226+0000 7f7b49ffb700 1 -- 172.21.15.103:0/2404284083 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7b3c009d90 con 0x7f7b4c133360 2023-07-02T09:04:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b49ffb700 1 -- 172.21.15.103:0/2404284083 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7b3c00e230 con 0x7f7b4c133360 2023-07-02T09:04:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b49ffb700 1 -- 172.21.15.103:0/2404284083 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7b3c00e4d0 con 0x7f7b4c133360 2023-07-02T09:04:18.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b51543700 1 -- 172.21.15.103:0/2404284083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c133360 msgr2=0x7f7b4c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:18.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b51543700 1 --2- 172.21.15.103:0/2404284083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c133360 0x7f7b4c09cc90 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:18.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b51543700 1 -- 172.21.15.103:0/2404284083 shutdown_connections 2023-07-02T09:04:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b51543700 1 --2- 172.21.15.103:0/2404284083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c133360 0x7f7b4c09cc90 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b51543700 1 -- 172.21.15.103:0/2404284083 >> 172.21.15.103:0/2404284083 conn(0x7f7b4c0a2080 msgr2=0x7f7b4c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b51543700 1 -- 172.21.15.103:0/2404284083 shutdown_connections 2023-07-02T09:04:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b51543700 1 -- 172.21.15.103:0/2404284083 wait complete. 2023-07-02T09:04:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.227+0000 7f7b51543700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.229+0000 7f7b51543700 1 Processor -- start 2023-07-02T09:04:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.229+0000 7f7b51543700 1 -- start start 2023-07-02T09:04:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.229+0000 7f7b51543700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c132f50 0x7f7b4c1ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.229+0000 7f7b51543700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c06d370 con 0x7f7b4c132f50 2023-07-02T09:04:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.229+0000 7f7b4affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c132f50 0x7f7b4c1ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.229+0000 7f7b4affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c132f50 0x7f7b4c1ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38236/0 (socket says 172.21.15.103:38236) 2023-07-02T09:04:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.229+0000 7f7b4affd700 1 -- 172.21.15.103:0/2445473703 learned_addr learned my addr 172.21.15.103:0/2445473703 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.230+0000 7f7b4affd700 1 -- 172.21.15.103:0/2445473703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c06dde0 con 0x7f7b4c132f50 2023-07-02T09:04:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.230+0000 7f7b4affd700 1 --2- 172.21.15.103:0/2445473703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c132f50 0x7f7b4c1ba9b0 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.230+0000 7f7b337fe700 1 -- 172.21.15.103:0/2445473703 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7b40004b80 con 0x7f7b4c132f50 2023-07-02T09:04:18.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.230+0000 7f7b337fe700 1 -- 172.21.15.103:0/2445473703 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f7b40004ce0 con 0x7f7b4c132f50 2023-07-02T09:04:18.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.230+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c1ba4f0 con 0x7f7b4c132f50 2023-07-02T09:04:18.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.230+0000 7f7b337fe700 1 -- 172.21.15.103:0/2445473703 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f7b4000c4f0 con 0x7f7b4c132f50 2023-07-02T09:04:18.234 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.230+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c071e40 con 0x7f7b4c132f50 2023-07-02T09:04:18.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.231+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c061ea0 con 0x7f7b4c132f50 2023-07-02T09:04:18.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.231+0000 7f7b337fe700 1 -- 172.21.15.103:0/2445473703 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f7b4000c650 con 0x7f7b4c132f50 2023-07-02T09:04:18.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.232+0000 7f7b337fe700 1 --2- 172.21.15.103:0/2445473703 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b380377c0 0x7f7b38039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:18.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.232+0000 7f7b337fe700 1 -- 172.21.15.103:0/2445473703 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f7b40046ba0 con 0x7f7b4c132f50 2023-07-02T09:04:18.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.232+0000 7f7b4a7fc700 1 --2- 172.21.15.103:0/2445473703 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b380377c0 0x7f7b38039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:18.235 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.232+0000 7f7b4a7fc700 1 --2- 172.21.15.103:0/2445473703 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b380377c0 0x7f7b38039cd0 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:18.237 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.236+0000 7f7b337fe700 1 -- 172.21.15.103:0/2445473703 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f7b4000acb0 con 0x7f7b4c132f50 2023-07-02T09:04:18.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.429+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7b4c0725b0 con 0x7f7b4c132f50 2023-07-02T09:04:18.430 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.429+0000 7f7b337fe700 1 -- 172.21.15.103:0/2445473703 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f7b4000a8d0 con 0x7f7b4c132f50 2023-07-02T09:04:18.431 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:18.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b380377c0 msgr2=0x7f7b38039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:18.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 --2- 172.21.15.103:0/2445473703 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b380377c0 0x7f7b38039cd0 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:18.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c132f50 msgr2=0x7f7b4c1ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:18.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 --2- 172.21.15.103:0/2445473703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c132f50 0x7f7b4c1ba9b0 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:18.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 shutdown_connections 2023-07-02T09:04:18.433 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 --2- 172.21.15.103:0/2445473703 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f7b380377c0 0x7f7b38039cd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:18.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 --2- 172.21.15.103:0/2445473703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b4c132f50 0x7f7b4c1ba9b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:18.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 >> 172.21.15.103:0/2445473703 conn(0x7f7b4c0a2080 msgr2=0x7f7b4c0a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:18.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 shutdown_connections 2023-07-02T09:04:18.434 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:18.431+0000 7f7b51543700 1 -- 172.21.15.103:0/2445473703 wait complete. 2023-07-02T09:04:18.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:19.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:19.591 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.585+0000 7f31ea748700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:19.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.590+0000 7f31ea748700 1 Processor -- start 2023-07-02T09:04:19.592 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.590+0000 7f31ea748700 1 -- start start 2023-07-02T09:04:19.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.591+0000 7f31ea748700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f31e4132f50 0x7f31e4133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:19.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.591+0000 7f31ea748700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e409e260 con 0x7f31e409c580 2023-07-02T09:04:19.593 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.591+0000 7f31ea748700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e40a1640 con 0x7f31e4132f50 2023-07-02T09:04:19.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.591+0000 7f31e3fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f31e4132f50 0x7f31e4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:19.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.591+0000 7f31e37fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f31e409c580 0x7f31e409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43736/0 (socket says 172.21.15.103:43736) 2023-07-02T09:04:19.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.591+0000 7f31e3fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f31e4132f50 0x7f31e4133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38248/0 (socket says 172.21.15.103:38248) 2023-07-02T09:04:19.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.591+0000 7f31e37fe700 1 -- 172.21.15.103:0/2526449441 learned_addr learned my addr 172.21.15.103:0/2526449441 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:19.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.592+0000 7f31e3fff700 1 -- 172.21.15.103:0/2526449441 >> v1:172.21.15.103:6789/0 conn(0x7f31e409c580 legacy=0x7f31e409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:19.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.592+0000 7f31e3fff700 1 -- 172.21.15.103:0/2526449441 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e4132a20 con 0x7f31e4132f50 2023-07-02T09:04:19.594 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.592+0000 7f31e3fff700 1 --2- 172.21.15.103:0/2526449441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e4132f50 0x7f31e4133360 crc :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f42c09c31e3d5f server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:19.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.592+0000 7f31e2ffd700 1 -- 172.21.15.103:0/2526449441 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f31d4009cd0 con 0x7f31e4132f50 2023-07-02T09:04:19.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.592+0000 7f31e2ffd700 1 -- 172.21.15.103:0/2526449441 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f31d4009e30 con 0x7f31e4132f50 2023-07-02T09:04:19.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.592+0000 7f31e2ffd700 1 -- 172.21.15.103:0/2526449441 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f31d400e360 con 0x7f31e4132f50 2023-07-02T09:04:19.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.592+0000 7f31ea748700 1 -- 172.21.15.103:0/2526449441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e4132f50 msgr2=0x7f31e4133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:19.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.592+0000 7f31ea748700 1 --2- 172.21.15.103:0/2526449441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e4132f50 0x7f31e4133360 crc :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:19.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.593+0000 7f31ea748700 1 -- 172.21.15.103:0/2526449441 shutdown_connections 2023-07-02T09:04:19.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.593+0000 7f31ea748700 1 --2- 172.21.15.103:0/2526449441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e4132f50 0x7f31e4133360 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:19.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.593+0000 7f31ea748700 1 -- 172.21.15.103:0/2526449441 >> 172.21.15.103:0/2526449441 conn(0x7f31e40a2080 msgr2=0x7f31e40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:19.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.593+0000 7f31ea748700 1 -- 172.21.15.103:0/2526449441 shutdown_connections 2023-07-02T09:04:19.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.593+0000 7f31ea748700 1 -- 172.21.15.103:0/2526449441 wait complete. 2023-07-02T09:04:19.597 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.593+0000 7f31ea748700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:19.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.594+0000 7f31ea748700 1 Processor -- start 2023-07-02T09:04:19.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.594+0000 7f31ea748700 1 -- start start 2023-07-02T09:04:19.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31ea748700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e409c580 0x7f31e412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:19.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31ea748700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e406f9d0 con 0x7f31e409c580 2023-07-02T09:04:19.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31e3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e409c580 0x7f31e412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:19.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31e3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e409c580 0x7f31e412b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38252/0 (socket says 172.21.15.103:38252) 2023-07-02T09:04:19.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31e3fff700 1 -- 172.21.15.103:0/1308785258 learned_addr learned my addr 172.21.15.103:0/1308785258 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:19.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31e3fff700 1 -- 172.21.15.103:0/1308785258 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e4133a00 con 0x7f31e409c580 2023-07-02T09:04:19.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31e3fff700 1 --2- 172.21.15.103:0/1308785258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e409c580 0x7f31e412b7b0 crc :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:19.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31e0ff9700 1 -- 172.21.15.103:0/1308785258 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f31d4013530 con 0x7f31e409c580 2023-07-02T09:04:19.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31e0ff9700 1 -- 172.21.15.103:0/1308785258 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f31d4013690 con 0x7f31e409c580 2023-07-02T09:04:19.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31e0ff9700 1 -- 172.21.15.103:0/1308785258 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f31d4013930 con 0x7f31e409c580 2023-07-02T09:04:19.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e406d160 con 0x7f31e409c580 2023-07-02T09:04:19.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.595+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e406da70 con 0x7f31e409c580 2023-07-02T09:04:19.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.596+0000 7f31e0ff9700 1 -- 172.21.15.103:0/1308785258 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f31d4013ae0 con 0x7f31e409c580 2023-07-02T09:04:19.600 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.596+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e4061ea0 con 0x7f31e409c580 2023-07-02T09:04:19.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.596+0000 7f31e0ff9700 1 --2- 172.21.15.103:0/1308785258 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31d0037810 0x7f31d0039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:19.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.596+0000 7f31e0ff9700 1 -- 172.21.15.103:0/1308785258 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f31d4018040 con 0x7f31e409c580 2023-07-02T09:04:19.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.597+0000 7f31e37fe700 1 --2- 172.21.15.103:0/1308785258 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31d0037810 0x7f31d0039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:19.601 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.597+0000 7f31e37fe700 1 --2- 172.21.15.103:0/1308785258 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31d0037810 0x7f31d0039d20 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:19.602 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.601+0000 7f31e0ff9700 1 -- 172.21.15.103:0/1308785258 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f31d4018dd0 con 0x7f31e409c580 2023-07-02T09:04:19.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.790+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f31e40725b0 con 0x7f31e409c580 2023-07-02T09:04:19.792 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.790+0000 7f31e0ff9700 1 -- 172.21.15.103:0/1308785258 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f31d40189f0 con 0x7f31e409c580 2023-07-02T09:04:19.792 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:19.794 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.792+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31d0037810 msgr2=0x7f31d0039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:19.794 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.792+0000 7f31ea748700 1 --2- 172.21.15.103:0/1308785258 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31d0037810 0x7f31d0039d20 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:19.794 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.792+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e409c580 msgr2=0x7f31e412b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:19.794 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.792+0000 7f31ea748700 1 --2- 172.21.15.103:0/1308785258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e409c580 0x7f31e412b7b0 crc :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:19.794 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.792+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 shutdown_connections 2023-07-02T09:04:19.794 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.792+0000 7f31ea748700 1 --2- 172.21.15.103:0/1308785258 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f31d0037810 0x7f31d0039d20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:19.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.792+0000 7f31ea748700 1 --2- 172.21.15.103:0/1308785258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31e409c580 0x7f31e412b7b0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:19.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.792+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 >> 172.21.15.103:0/1308785258 conn(0x7f31e40a2080 msgr2=0x7f31e41babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:19.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.793+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 shutdown_connections 2023-07-02T09:04:19.795 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:19.793+0000 7f31ea748700 1 -- 172.21.15.103:0/1308785258 wait complete. 2023-07-02T09:04:19.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:20.808 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:20.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.948+0000 7f36f1da2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:20.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.953+0000 7f36f1da2700 1 Processor -- start 2023-07-02T09:04:20.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.953+0000 7f36f1da2700 1 -- start start 2023-07-02T09:04:20.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36f1da2700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36ec132f50 0x7f36ec133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:20.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36f1da2700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec09e260 con 0x7f36ec09c580 2023-07-02T09:04:20.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36f1da2700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36ec0a1640 con 0x7f36ec132f50 2023-07-02T09:04:20.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36eb7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36ec132f50 0x7f36ec133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:20.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36eaffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f36ec09c580 0x7f36ec09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43738/0 (socket says 172.21.15.103:43738) 2023-07-02T09:04:20.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36eb7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f36ec132f50 0x7f36ec133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38260/0 (socket says 172.21.15.103:38260) 2023-07-02T09:04:20.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36eaffd700 1 -- 172.21.15.103:0/3131966481 learned_addr learned my addr 172.21.15.103:0/3131966481 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:20.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36eb7fe700 1 -- 172.21.15.103:0/3131966481 >> v1:172.21.15.103:6789/0 conn(0x7f36ec09c580 legacy=0x7f36ec09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:20.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36eb7fe700 1 -- 172.21.15.103:0/3131966481 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec132a20 con 0x7f36ec132f50 2023-07-02T09:04:20.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.954+0000 7f36ea7fc700 1 -- 172.21.15.103:0/3131966481 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413374918 0 0) 0x7f36ec09e260 con 0x7f36ec09c580 2023-07-02T09:04:20.957 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36eb7fe700 1 --2- 172.21.15.103:0/3131966481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec132f50 0x7f36ec133360 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff07b46ca9b6720 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:20.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36ea7fc700 1 -- 172.21.15.103:0/3131966481 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36d4009cd0 con 0x7f36ec132f50 2023-07-02T09:04:20.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36ea7fc700 1 -- 172.21.15.103:0/3131966481 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f36d4009e30 con 0x7f36ec132f50 2023-07-02T09:04:20.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36ea7fc700 1 -- 172.21.15.103:0/3131966481 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36d400e360 con 0x7f36ec132f50 2023-07-02T09:04:20.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36f1da2700 1 -- 172.21.15.103:0/3131966481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec132f50 msgr2=0x7f36ec133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:20.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36f1da2700 1 --2- 172.21.15.103:0/3131966481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec132f50 0x7f36ec133360 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:20.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36f1da2700 1 -- 172.21.15.103:0/3131966481 shutdown_connections 2023-07-02T09:04:20.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36f1da2700 1 --2- 172.21.15.103:0/3131966481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec132f50 0x7f36ec133360 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:20.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.955+0000 7f36f1da2700 1 -- 172.21.15.103:0/3131966481 >> 172.21.15.103:0/3131966481 conn(0x7f36ec0a2080 msgr2=0x7f36ec0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:20.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.956+0000 7f36f1da2700 1 -- 172.21.15.103:0/3131966481 shutdown_connections 2023-07-02T09:04:20.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.956+0000 7f36f1da2700 1 -- 172.21.15.103:0/3131966481 wait complete. 2023-07-02T09:04:20.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.956+0000 7f36f1da2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:20.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.957+0000 7f36f1da2700 1 Processor -- start 2023-07-02T09:04:20.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.957+0000 7f36f1da2700 1 -- start start 2023-07-02T09:04:20.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.957+0000 7f36f1da2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec09c580 0x7f36ec1baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:20.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.957+0000 7f36f1da2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec1ba610 con 0x7f36ec09c580 2023-07-02T09:04:20.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36eb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec09c580 0x7f36ec1baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:20.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36eb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec09c580 0x7f36ec1baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38262/0 (socket says 172.21.15.103:38262) 2023-07-02T09:04:20.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36eb7fe700 1 -- 172.21.15.103:0/2008341387 learned_addr learned my addr 172.21.15.103:0/2008341387 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:20.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36eb7fe700 1 -- 172.21.15.103:0/2008341387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec06dfc0 con 0x7f36ec09c580 2023-07-02T09:04:20.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36eb7fe700 1 --2- 172.21.15.103:0/2008341387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec09c580 0x7f36ec1baa60 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:20.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36e3fff700 1 -- 172.21.15.103:0/2008341387 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36d4013530 con 0x7f36ec09c580 2023-07-02T09:04:20.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36e3fff700 1 -- 172.21.15.103:0/2008341387 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f36d4013690 con 0x7f36ec09c580 2023-07-02T09:04:20.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec1ba270 con 0x7f36ec09c580 2023-07-02T09:04:20.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36e3fff700 1 -- 172.21.15.103:0/2008341387 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f36d4013930 con 0x7f36ec09c580 2023-07-02T09:04:20.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.958+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec06dba0 con 0x7f36ec09c580 2023-07-02T09:04:20.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.959+0000 7f36e3fff700 1 -- 172.21.15.103:0/2008341387 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f36d4013b30 con 0x7f36ec09c580 2023-07-02T09:04:20.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.959+0000 7f36e3fff700 1 --2- 172.21.15.103:0/2008341387 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36cc037760 0x7f36cc039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:20.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.959+0000 7f36e3fff700 1 -- 172.21.15.103:0/2008341387 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f36d4018040 con 0x7f36ec09c580 2023-07-02T09:04:20.963 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.959+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec1ba460 con 0x7f36ec09c580 2023-07-02T09:04:20.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.959+0000 7f36eaffd700 1 --2- 172.21.15.103:0/2008341387 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36cc037760 0x7f36cc039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:20.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.960+0000 7f36eaffd700 1 --2- 172.21.15.103:0/2008341387 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36cc037760 0x7f36cc039c70 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:20.965 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:20.964+0000 7f36e3fff700 1 -- 172.21.15.103:0/2008341387 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f36d4013e30 con 0x7f36ec09c580 2023-07-02T09:04:21.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~275s 2023-07-02T09:04:21.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.155+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f36ec061ea0 con 0x7f36ec09c580 2023-07-02T09:04:21.156 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.155+0000 7f36e3fff700 1 -- 172.21.15.103:0/2008341387 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f36d4010e60 con 0x7f36ec09c580 2023-07-02T09:04:21.157 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:21.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.157+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36cc037760 msgr2=0x7f36cc039c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:21.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.157+0000 7f36f1da2700 1 --2- 172.21.15.103:0/2008341387 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36cc037760 0x7f36cc039c70 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:21.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.157+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec09c580 msgr2=0x7f36ec1baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:21.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.157+0000 7f36f1da2700 1 --2- 172.21.15.103:0/2008341387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec09c580 0x7f36ec1baa60 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:21.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.157+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 shutdown_connections 2023-07-02T09:04:21.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.157+0000 7f36f1da2700 1 --2- 172.21.15.103:0/2008341387 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f36cc037760 0x7f36cc039c70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:21.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.157+0000 7f36f1da2700 1 --2- 172.21.15.103:0/2008341387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36ec09c580 0x7f36ec1baa60 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:21.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.157+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 >> 172.21.15.103:0/2008341387 conn(0x7f36ec0a2080 msgr2=0x7f36ec130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:21.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.158+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 shutdown_connections 2023-07-02T09:04:21.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:21.158+0000 7f36f1da2700 1 -- 172.21.15.103:0/2008341387 wait complete. 2023-07-02T09:04:21.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:22.173 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:22.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.310+0000 7f6740b2c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:22.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.314+0000 7f6740b2c700 1 Processor -- start 2023-07-02T09:04:22.316 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.315+0000 7f6740b2c700 1 -- start start 2023-07-02T09:04:22.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.315+0000 7f6740b2c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f673c12f250 0x7f673c12f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:22.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.315+0000 7f6740b2c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c06df60 con 0x7f673c098a00 2023-07-02T09:04:22.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.315+0000 7f6740b2c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c09a710 con 0x7f673c12f250 2023-07-02T09:04:22.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673a59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f673c12f250 0x7f673c12f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:22.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f6739d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f673c098a00 0x7f673c098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:43754/0 (socket says 172.21.15.103:43754) 2023-07-02T09:04:22.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673a59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f673c12f250 0x7f673c12f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38278/0 (socket says 172.21.15.103:38278) 2023-07-02T09:04:22.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f6739d9b700 1 -- 172.21.15.103:0/2439674558 learned_addr learned my addr 172.21.15.103:0/2439674558 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:22.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673a59c700 1 -- 172.21.15.103:0/2439674558 >> v1:172.21.15.103:6789/0 conn(0x7f673c098a00 legacy=0x7f673c098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:22.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673a59c700 1 -- 172.21.15.103:0/2439674558 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c12ed20 con 0x7f673c12f250 2023-07-02T09:04:22.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673959a700 1 -- 172.21.15.103:0/2439674558 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992060417 0 0) 0x7f673c06df60 con 0x7f673c098a00 2023-07-02T09:04:22.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673a59c700 1 --2- 172.21.15.103:0/2439674558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c12f250 0x7f673c12f660 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a07385277eb8d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:22.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673959a700 1 -- 172.21.15.103:0/2439674558 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f672400ac60 con 0x7f673c12f250 2023-07-02T09:04:22.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673959a700 1 -- 172.21.15.103:0/2439674558 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f672400adc0 con 0x7f673c12f250 2023-07-02T09:04:22.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.316+0000 7f673959a700 1 -- 172.21.15.103:0/2439674558 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f672400f470 con 0x7f673c12f250 2023-07-02T09:04:22.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.317+0000 7f6740b2c700 1 -- 172.21.15.103:0/2439674558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c12f250 msgr2=0x7f673c12f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:22.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.317+0000 7f6740b2c700 1 --2- 172.21.15.103:0/2439674558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c12f250 0x7f673c12f660 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:22.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.317+0000 7f6740b2c700 1 -- 172.21.15.103:0/2439674558 shutdown_connections 2023-07-02T09:04:22.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.317+0000 7f6740b2c700 1 --2- 172.21.15.103:0/2439674558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c12f250 0x7f673c12f660 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:22.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.317+0000 7f6740b2c700 1 -- 172.21.15.103:0/2439674558 >> 172.21.15.103:0/2439674558 conn(0x7f673c09d810 msgr2=0x7f673c09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:22.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.317+0000 7f6740b2c700 1 -- 172.21.15.103:0/2439674558 shutdown_connections 2023-07-02T09:04:22.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.317+0000 7f6740b2c700 1 -- 172.21.15.103:0/2439674558 wait complete. 2023-07-02T09:04:22.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.317+0000 7f6740b2c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:22.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.319+0000 7f6740b2c700 1 Processor -- start 2023-07-02T09:04:22.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.319+0000 7f6740b2c700 1 -- start start 2023-07-02T09:04:22.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.319+0000 7f6740b2c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c098a00 0x7f673c117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:22.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.319+0000 7f6740b2c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c118750 con 0x7f673c098a00 2023-07-02T09:04:22.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f673a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c098a00 0x7f673c117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:22.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f673a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c098a00 0x7f673c117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38292/0 (socket says 172.21.15.103:38292) 2023-07-02T09:04:22.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f673a59c700 1 -- 172.21.15.103:0/342261712 learned_addr learned my addr 172.21.15.103:0/342261712 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:22.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f673a59c700 1 -- 172.21.15.103:0/342261712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c11a450 con 0x7f673c098a00 2023-07-02T09:04:22.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f673a59c700 1 --2- 172.21.15.103:0/342261712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c098a00 0x7f673c117360 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:22.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f6732ffd700 1 -- 172.21.15.103:0/342261712 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67240103b0 con 0x7f673c098a00 2023-07-02T09:04:22.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f6732ffd700 1 -- 172.21.15.103:0/342261712 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6724010510 con 0x7f673c098a00 2023-07-02T09:04:22.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c118e20 con 0x7f673c098a00 2023-07-02T09:04:22.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f6732ffd700 1 -- 172.21.15.103:0/342261712 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f67240107b0 con 0x7f673c098a00 2023-07-02T09:04:22.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.320+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c119e70 con 0x7f673c098a00 2023-07-02T09:04:22.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.321+0000 7f6732ffd700 1 -- 172.21.15.103:0/342261712 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f6724010960 con 0x7f673c098a00 2023-07-02T09:04:22.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.321+0000 7f6732ffd700 1 --2- 172.21.15.103:0/342261712 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f671c0377c0 0x7f671c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:22.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.321+0000 7f6732ffd700 1 -- 172.21.15.103:0/342261712 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f6724017040 con 0x7f673c098a00 2023-07-02T09:04:22.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.321+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c061ea0 con 0x7f673c098a00 2023-07-02T09:04:22.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.322+0000 7f6739d9b700 1 --2- 172.21.15.103:0/342261712 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f671c0377c0 0x7f671c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:22.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.322+0000 7f6739d9b700 1 --2- 172.21.15.103:0/342261712 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f671c0377c0 0x7f671c039cd0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:22.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.326+0000 7f6732ffd700 1 -- 172.21.15.103:0/342261712 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f6724017200 con 0x7f673c098a00 2023-07-02T09:04:22.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.518+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f673c073ca0 con 0x7f673c098a00 2023-07-02T09:04:22.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.518+0000 7f6732ffd700 1 -- 172.21.15.103:0/342261712 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f6724017200 con 0x7f673c098a00 2023-07-02T09:04:22.520 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f671c0377c0 msgr2=0x7f671c039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 --2- 172.21.15.103:0/342261712 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f671c0377c0 0x7f671c039cd0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c098a00 msgr2=0x7f673c117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 --2- 172.21.15.103:0/342261712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c098a00 0x7f673c117360 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 shutdown_connections 2023-07-02T09:04:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 --2- 172.21.15.103:0/342261712 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f671c0377c0 0x7f671c039cd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 --2- 172.21.15.103:0/342261712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f673c098a00 0x7f673c117360 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 >> 172.21.15.103:0/342261712 conn(0x7f673c09d810 msgr2=0x7f673c0a0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 shutdown_connections 2023-07-02T09:04:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:22.520+0000 7f6740b2c700 1 -- 172.21.15.103:0/342261712 wait complete. 2023-07-02T09:04:22.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:23.535 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:23.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.672+0000 7f922747b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:23.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.677+0000 7f922747b700 1 Processor -- start 2023-07-02T09:04:23.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.677+0000 7f922747b700 1 -- start start 2023-07-02T09:04:23.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.678+0000 7f922747b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9220133360 0x7f922009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:23.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.678+0000 7f922747b700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922009e260 con 0x7f9220132f50 2023-07-02T09:04:23.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.678+0000 7f922747b700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92200a1640 con 0x7f9220133360 2023-07-02T09:04:23.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.678+0000 7f9224a16700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9220133360 0x7f922009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:23.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.678+0000 7f9225217700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9220132f50 0x7f922009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46168/0 (socket says 172.21.15.103:46168) 2023-07-02T09:04:23.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.678+0000 7f9224a16700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9220133360 0x7f922009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52468/0 (socket says 172.21.15.103:52468) 2023-07-02T09:04:23.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.678+0000 7f9225217700 1 -- 172.21.15.103:0/3805850779 learned_addr learned my addr 172.21.15.103:0/3805850779 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:23.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.679+0000 7f9224a16700 1 -- 172.21.15.103:0/3805850779 >> v1:172.21.15.103:6789/0 conn(0x7f9220132f50 legacy=0x7f922009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.679+0000 7f9224a16700 1 -- 172.21.15.103:0/3805850779 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220132a20 con 0x7f9220133360 2023-07-02T09:04:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.679+0000 7f9217fff700 1 -- 172.21.15.103:0/3805850779 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197579364 0 0) 0x7f922009e260 con 0x7f9220132f50 2023-07-02T09:04:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.679+0000 7f9224a16700 1 --2- 172.21.15.103:0/3805850779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220133360 0x7f922009cc90 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24372e0fb485e76d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.679+0000 7f9217fff700 1 -- 172.21.15.103:0/3805850779 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f921c009cd0 con 0x7f9220133360 2023-07-02T09:04:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.679+0000 7f9217fff700 1 -- 172.21.15.103:0/3805850779 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f921c009e30 con 0x7f9220133360 2023-07-02T09:04:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.679+0000 7f9217fff700 1 -- 172.21.15.103:0/3805850779 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f921c00e360 con 0x7f9220133360 2023-07-02T09:04:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.680+0000 7f922747b700 1 -- 172.21.15.103:0/3805850779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220133360 msgr2=0x7f922009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.680+0000 7f922747b700 1 --2- 172.21.15.103:0/3805850779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220133360 0x7f922009cc90 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.680+0000 7f922747b700 1 -- 172.21.15.103:0/3805850779 shutdown_connections 2023-07-02T09:04:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.680+0000 7f922747b700 1 --2- 172.21.15.103:0/3805850779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220133360 0x7f922009cc90 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.680+0000 7f922747b700 1 -- 172.21.15.103:0/3805850779 >> 172.21.15.103:0/3805850779 conn(0x7f92200a2080 msgr2=0x7f92200a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.680+0000 7f922747b700 1 -- 172.21.15.103:0/3805850779 shutdown_connections 2023-07-02T09:04:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.680+0000 7f922747b700 1 -- 172.21.15.103:0/3805850779 wait complete. 2023-07-02T09:04:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.680+0000 7f922747b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:23.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.681+0000 7f922747b700 1 Processor -- start 2023-07-02T09:04:23.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.681+0000 7f922747b700 1 -- start start 2023-07-02T09:04:23.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.681+0000 7f922747b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220132f50 0x7f92201ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:23.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.681+0000 7f922747b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922006d370 con 0x7f9220132f50 2023-07-02T09:04:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f9225217700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220132f50 0x7f92201ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f9225217700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220132f50 0x7f92201ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52472/0 (socket says 172.21.15.103:52472) 2023-07-02T09:04:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f9225217700 1 -- 172.21.15.103:0/1628723254 learned_addr learned my addr 172.21.15.103:0/1628723254 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f9225217700 1 -- 172.21.15.103:0/1628723254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922006dde0 con 0x7f9220132f50 2023-07-02T09:04:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f9225217700 1 --2- 172.21.15.103:0/1628723254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220132f50 0x7f92201ba9b0 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f9215ffb700 1 -- 172.21.15.103:0/1628723254 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9210004b80 con 0x7f9220132f50 2023-07-02T09:04:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f9215ffb700 1 -- 172.21.15.103:0/1628723254 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9210004ce0 con 0x7f9220132f50 2023-07-02T09:04:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92201ba4f0 con 0x7f9220132f50 2023-07-02T09:04:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f9215ffb700 1 -- 172.21.15.103:0/1628723254 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f921000c4f0 con 0x7f9220132f50 2023-07-02T09:04:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.682+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9220071e40 con 0x7f9220132f50 2023-07-02T09:04:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.683+0000 7f9215ffb700 1 -- 172.21.15.103:0/1628723254 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f921000c650 con 0x7f9220132f50 2023-07-02T09:04:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.683+0000 7f9215ffb700 1 --2- 172.21.15.103:0/1628723254 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92040377c0 0x7f9204039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.683+0000 7f9215ffb700 1 -- 172.21.15.103:0/1628723254 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f9210046970 con 0x7f9220132f50 2023-07-02T09:04:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.683+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9220061ea0 con 0x7f9220132f50 2023-07-02T09:04:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.683+0000 7f9224a16700 1 --2- 172.21.15.103:0/1628723254 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92040377c0 0x7f9204039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:23.688 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.684+0000 7f9224a16700 1 --2- 172.21.15.103:0/1628723254 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92040377c0 0x7f9204039cd0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:23.689 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.688+0000 7f9215ffb700 1 -- 172.21.15.103:0/1628723254 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f9210009c80 con 0x7f9220132f50 2023-07-02T09:04:23.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.880+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f92200725b0 con 0x7f9220132f50 2023-07-02T09:04:23.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.880+0000 7f9215ffb700 1 -- 172.21.15.103:0/1628723254 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f921002c020 con 0x7f9220132f50 2023-07-02T09:04:23.882 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:23.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92040377c0 msgr2=0x7f9204039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:23.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 --2- 172.21.15.103:0/1628723254 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92040377c0 0x7f9204039cd0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:23.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220132f50 msgr2=0x7f92201ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:23.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 --2- 172.21.15.103:0/1628723254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220132f50 0x7f92201ba9b0 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 shutdown_connections 2023-07-02T09:04:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 --2- 172.21.15.103:0/1628723254 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92040377c0 0x7f9204039cd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 --2- 172.21.15.103:0/1628723254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9220132f50 0x7f92201ba9b0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 >> 172.21.15.103:0/1628723254 conn(0x7f92200a2080 msgr2=0x7f92200a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 shutdown_connections 2023-07-02T09:04:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:23.883+0000 7f922747b700 1 -- 172.21.15.103:0/1628723254 wait complete. 2023-07-02T09:04:23.897 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:24.897 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:25.042 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.038+0000 7f92d322e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:25.043 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.042+0000 7f92d322e700 1 Processor -- start 2023-07-02T09:04:25.043 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.042+0000 7f92d322e700 1 -- start start 2023-07-02T09:04:25.044 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92d322e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f92cc132030 0x7f92cc134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:25.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92d322e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc06df60 con 0x7f92cc12f250 2023-07-02T09:04:25.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92d322e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc09a710 con 0x7f92cc132030 2023-07-02T09:04:25.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cbfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f92cc132030 0x7f92cc134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:25.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cbfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f92cc132030 0x7f92cc134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52478/0 (socket says 172.21.15.103:52478) 2023-07-02T09:04:25.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cbfff700 1 -- 172.21.15.103:0/83628974 learned_addr learned my addr 172.21.15.103:0/83628974 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:25.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92d0fca700 1 --1- 172.21.15.103:0/83628974 >> v1:172.21.15.103:6789/0 conn(0x7f92cc12f250 0x7f92cc12f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46184/0 (socket says 172.21.15.103:46184) 2023-07-02T09:04:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cbfff700 1 -- 172.21.15.103:0/83628974 >> v1:172.21.15.103:6789/0 conn(0x7f92cc12f250 legacy=0x7f92cc12f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cb7fe700 1 -- 172.21.15.103:0/83628974 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723686132 0 0) 0x7f92cc06df60 con 0x7f92cc12f250 2023-07-02T09:04:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cbfff700 1 -- 172.21.15.103:0/83628974 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc12ed20 con 0x7f92cc132030 2023-07-02T09:04:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cbfff700 1 --2- 172.21.15.103:0/83628974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc132030 0x7f92cc134450 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a056886ca0d5a685 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cb7fe700 1 -- 172.21.15.103:0/83628974 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f92b8009d90 con 0x7f92cc132030 2023-07-02T09:04:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cb7fe700 1 -- 172.21.15.103:0/83628974 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f92b800e230 con 0x7f92cc132030 2023-07-02T09:04:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.043+0000 7f92cb7fe700 1 -- 172.21.15.103:0/83628974 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f92b800e4d0 con 0x7f92cc132030 2023-07-02T09:04:25.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.044+0000 7f92d322e700 1 -- 172.21.15.103:0/83628974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc132030 msgr2=0x7f92cc134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:25.048 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.044+0000 7f92d322e700 1 --2- 172.21.15.103:0/83628974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc132030 0x7f92cc134450 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.044+0000 7f92d322e700 1 -- 172.21.15.103:0/83628974 shutdown_connections 2023-07-02T09:04:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.044+0000 7f92d322e700 1 --2- 172.21.15.103:0/83628974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc132030 0x7f92cc134450 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.044+0000 7f92d322e700 1 -- 172.21.15.103:0/83628974 >> 172.21.15.103:0/83628974 conn(0x7f92cc09d810 msgr2=0x7f92cc09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.044+0000 7f92d322e700 1 -- 172.21.15.103:0/83628974 shutdown_connections 2023-07-02T09:04:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.044+0000 7f92d322e700 1 -- 172.21.15.103:0/83628974 wait complete. 2023-07-02T09:04:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.044+0000 7f92d322e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.045+0000 7f92d322e700 1 Processor -- start 2023-07-02T09:04:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.045+0000 7f92d322e700 1 -- start start 2023-07-02T09:04:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.046+0000 7f92d322e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc12f250 0x7f92cc117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.046+0000 7f92d322e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc118750 con 0x7f92cc12f250 2023-07-02T09:04:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.046+0000 7f92d0fca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc12f250 0x7f92cc117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.046+0000 7f92d0fca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc12f250 0x7f92cc117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52482/0 (socket says 172.21.15.103:52482) 2023-07-02T09:04:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.046+0000 7f92d0fca700 1 -- 172.21.15.103:0/190062471 learned_addr learned my addr 172.21.15.103:0/190062471 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.046+0000 7f92d0fca700 1 -- 172.21.15.103:0/190062471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc11a3c0 con 0x7f92cc12f250 2023-07-02T09:04:25.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.047+0000 7f92d0fca700 1 --2- 172.21.15.103:0/190062471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc12f250 0x7f92cc117360 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:25.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.047+0000 7f92c97fa700 1 -- 172.21.15.103:0/190062471 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f92c0004b80 con 0x7f92cc12f250 2023-07-02T09:04:25.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.047+0000 7f92c97fa700 1 -- 172.21.15.103:0/190062471 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f92c0004ce0 con 0x7f92cc12f250 2023-07-02T09:04:25.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.047+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc119990 con 0x7f92cc12f250 2023-07-02T09:04:25.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.047+0000 7f92c97fa700 1 -- 172.21.15.103:0/190062471 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f92c000c4f0 con 0x7f92cc12f250 2023-07-02T09:04:25.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.047+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc119ce0 con 0x7f92cc12f250 2023-07-02T09:04:25.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.048+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc061ea0 con 0x7f92cc12f250 2023-07-02T09:04:25.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.048+0000 7f92c97fa700 1 -- 172.21.15.103:0/190062471 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f92c000c650 con 0x7f92cc12f250 2023-07-02T09:04:25.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.048+0000 7f92c97fa700 1 --2- 172.21.15.103:0/190062471 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92b00377c0 0x7f92b0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:25.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.048+0000 7f92c97fa700 1 -- 172.21.15.103:0/190062471 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f92c00469e0 con 0x7f92cc12f250 2023-07-02T09:04:25.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.048+0000 7f92cbfff700 1 --2- 172.21.15.103:0/190062471 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92b00377c0 0x7f92b0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:25.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.049+0000 7f92cbfff700 1 --2- 172.21.15.103:0/190062471 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92b00377c0 0x7f92b0039cd0 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:25.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.053+0000 7f92c97fa700 1 -- 172.21.15.103:0/190062471 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f92c0009c80 con 0x7f92cc12f250 2023-07-02T09:04:25.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.243+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f92cc12c6c0 con 0x7f92cc12f250 2023-07-02T09:04:25.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.243+0000 7f92c97fa700 1 -- 172.21.15.103:0/190062471 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f92c00098a0 con 0x7f92cc12f250 2023-07-02T09:04:25.245 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:25.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.245+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92b00377c0 msgr2=0x7f92b0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:25.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.245+0000 7f92d322e700 1 --2- 172.21.15.103:0/190062471 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92b00377c0 0x7f92b0039cd0 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:25.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.246+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc12f250 msgr2=0x7f92cc117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:25.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.246+0000 7f92d322e700 1 --2- 172.21.15.103:0/190062471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc12f250 0x7f92cc117360 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:25.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.246+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 shutdown_connections 2023-07-02T09:04:25.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.246+0000 7f92d322e700 1 --2- 172.21.15.103:0/190062471 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f92b00377c0 0x7f92b0039cd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:25.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.246+0000 7f92d322e700 1 --2- 172.21.15.103:0/190062471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc12f250 0x7f92cc117360 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:25.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.246+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 >> 172.21.15.103:0/190062471 conn(0x7f92cc09d810 msgr2=0x7f92cc12dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:25.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.246+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 shutdown_connections 2023-07-02T09:04:25.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:25.246+0000 7f92d322e700 1 -- 172.21.15.103:0/190062471 wait complete. 2023-07-02T09:04:25.260 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:26.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:26.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.395+0000 7f5309c1f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.401+0000 7f5309c1f700 1 Processor -- start 2023-07-02T09:04:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.401+0000 7f5309c1f700 1 -- start start 2023-07-02T09:04:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.401+0000 7f5309c1f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5304132030 0x7f5304134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.401+0000 7f5309c1f700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530406df60 con 0x7f530412f250 2023-07-02T09:04:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.401+0000 7f5309c1f700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530409a710 con 0x7f5304132030 2023-07-02T09:04:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.401+0000 7f5302ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5304132030 0x7f5304134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f53037fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f530412f250 0x7f530412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46190/0 (socket says 172.21.15.103:46190) 2023-07-02T09:04:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f5302ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5304132030 0x7f5304134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52488/0 (socket says 172.21.15.103:52488) 2023-07-02T09:04:26.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f53037fe700 1 -- 172.21.15.103:0/1450845918 learned_addr learned my addr 172.21.15.103:0/1450845918 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:26.404 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f5302ffd700 1 -- 172.21.15.103:0/1450845918 >> v1:172.21.15.103:6789/0 conn(0x7f530412f250 legacy=0x7f530412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:26.404 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f5302ffd700 1 -- 172.21.15.103:0/1450845918 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530412ed20 con 0x7f5304132030 2023-07-02T09:04:26.404 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f53027fc700 1 -- 172.21.15.103:0/1450845918 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881184711 0 0) 0x7f530406df60 con 0x7f530412f250 2023-07-02T09:04:26.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f5302ffd700 1 --2- 172.21.15.103:0/1450845918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5304132030 0x7f5304134450 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40719cbcc40d0cb0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:26.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f53027fc700 1 -- 172.21.15.103:0/1450845918 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f52f8009cd0 con 0x7f5304132030 2023-07-02T09:04:26.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f53027fc700 1 -- 172.21.15.103:0/1450845918 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f52f8009e30 con 0x7f5304132030 2023-07-02T09:04:26.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.402+0000 7f53027fc700 1 -- 172.21.15.103:0/1450845918 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f52f800e360 con 0x7f5304132030 2023-07-02T09:04:26.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.403+0000 7f5309c1f700 1 -- 172.21.15.103:0/1450845918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5304132030 msgr2=0x7f5304134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:26.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.403+0000 7f5309c1f700 1 --2- 172.21.15.103:0/1450845918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5304132030 0x7f5304134450 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:26.405 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.403+0000 7f5309c1f700 1 -- 172.21.15.103:0/1450845918 shutdown_connections 2023-07-02T09:04:26.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.403+0000 7f5309c1f700 1 --2- 172.21.15.103:0/1450845918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5304132030 0x7f5304134450 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:26.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.403+0000 7f5309c1f700 1 -- 172.21.15.103:0/1450845918 >> 172.21.15.103:0/1450845918 conn(0x7f530409d810 msgr2=0x7f530409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:26.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.403+0000 7f5309c1f700 1 -- 172.21.15.103:0/1450845918 shutdown_connections 2023-07-02T09:04:26.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.403+0000 7f5309c1f700 1 -- 172.21.15.103:0/1450845918 wait complete. 2023-07-02T09:04:26.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.403+0000 7f5309c1f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:26.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.404+0000 7f5309c1f700 1 Processor -- start 2023-07-02T09:04:26.406 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.404+0000 7f5309c1f700 1 -- start start 2023-07-02T09:04:26.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f5309c1f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f530412f250 0x7f530412b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:26.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f5309c1f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530411a010 con 0x7f530412f250 2023-07-02T09:04:26.408 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f53037fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f530412f250 0x7f530412b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:26.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f53037fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f530412f250 0x7f530412b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52492/0 (socket says 172.21.15.103:52492) 2023-07-02T09:04:26.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f53037fe700 1 -- 172.21.15.103:0/1701036454 learned_addr learned my addr 172.21.15.103:0/1701036454 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:26.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f53037fe700 1 -- 172.21.15.103:0/1701036454 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5304070cf0 con 0x7f530412f250 2023-07-02T09:04:26.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f53037fe700 1 --2- 172.21.15.103:0/1701036454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f530412f250 0x7f530412b5a0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:26.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f52ebfff700 1 -- 172.21.15.103:0/1701036454 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f52f400ebe0 con 0x7f530412f250 2023-07-02T09:04:26.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f52ebfff700 1 -- 172.21.15.103:0/1701036454 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f52f400ed40 con 0x7f530412f250 2023-07-02T09:04:26.409 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f52ebfff700 1 -- 172.21.15.103:0/1701036454 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f52f40070c0 con 0x7f530412f250 2023-07-02T09:04:26.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.405+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5304070f50 con 0x7f530412f250 2023-07-02T09:04:26.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.406+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5304118720 con 0x7f530412f250 2023-07-02T09:04:26.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.406+0000 7f52ebfff700 1 -- 172.21.15.103:0/1701036454 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f52f4007220 con 0x7f530412f250 2023-07-02T09:04:26.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.406+0000 7f52ebfff700 1 --2- 172.21.15.103:0/1701036454 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52f00377c0 0x7f52f0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:26.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.406+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5304071140 con 0x7f530412f250 2023-07-02T09:04:26.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.406+0000 7f52ebfff700 1 -- 172.21.15.103:0/1701036454 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f52f40496b0 con 0x7f530412f250 2023-07-02T09:04:26.410 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.407+0000 7f5302ffd700 1 --2- 172.21.15.103:0/1701036454 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52f00377c0 0x7f52f0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:26.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.407+0000 7f5302ffd700 1 --2- 172.21.15.103:0/1701036454 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52f00377c0 0x7f52f0039cd0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:26.411 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.411+0000 7f52ebfff700 1 -- 172.21.15.103:0/1701036454 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f52f4007520 con 0x7f530412f250 2023-07-02T09:04:26.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2023-07-02T09:04:26.603 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.603+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5304061ea0 con 0x7f530412f250 2023-07-02T09:04:26.604 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.604+0000 7f52ebfff700 1 -- 172.21.15.103:0/1701036454 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f52f400a250 con 0x7f530412f250 2023-07-02T09:04:26.604 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:26.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52f00377c0 msgr2=0x7f52f0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:26.606 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 --2- 172.21.15.103:0/1701036454 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52f00377c0 0x7f52f0039cd0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:26.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f530412f250 msgr2=0x7f530412b5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:26.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 --2- 172.21.15.103:0/1701036454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f530412f250 0x7f530412b5a0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:26.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 shutdown_connections 2023-07-02T09:04:26.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 --2- 172.21.15.103:0/1701036454 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f52f00377c0 0x7f52f0039cd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:26.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 --2- 172.21.15.103:0/1701036454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f530412f250 0x7f530412b5a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:26.607 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 >> 172.21.15.103:0/1701036454 conn(0x7f530409d810 msgr2=0x7f530412e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:26.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 shutdown_connections 2023-07-02T09:04:26.608 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:26.606+0000 7f5309c1f700 1 -- 172.21.15.103:0/1701036454 wait complete. 2023-07-02T09:04:26.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:27.620 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:27.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.760+0000 7fac59523700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.765+0000 7fac59523700 1 Processor -- start 2023-07-02T09:04:27.766 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.766+0000 7fac59523700 1 -- start start 2023-07-02T09:04:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.766+0000 7fac59523700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fac54133360 0x7fac5409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.766+0000 7fac59523700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5409e260 con 0x7fac54132f50 2023-07-02T09:04:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.766+0000 7fac59523700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac540a1640 con 0x7fac54133360 2023-07-02T09:04:27.767 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.766+0000 7fac527fc700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fac54133360 0x7fac5409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.766+0000 7fac52ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fac54132f50 0x7fac5409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46198/0 (socket says 172.21.15.103:46198) 2023-07-02T09:04:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.766+0000 7fac52ffd700 1 -- 172.21.15.103:0/4192539312 learned_addr learned my addr 172.21.15.103:0/4192539312 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.767+0000 7fac527fc700 1 -- 172.21.15.103:0/4192539312 >> v1:172.21.15.103:6789/0 conn(0x7fac54132f50 legacy=0x7fac5409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.767+0000 7fac527fc700 1 -- 172.21.15.103:0/4192539312 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac54132a20 con 0x7fac54133360 2023-07-02T09:04:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.767+0000 7fac51ffb700 1 -- 172.21.15.103:0/4192539312 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707671410 0 0) 0x7fac5409e260 con 0x7fac54132f50 2023-07-02T09:04:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.767+0000 7fac527fc700 1 --2- 172.21.15.103:0/4192539312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54133360 0x7fac5409cc90 crc :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25193671a0267aa9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:27.768 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.767+0000 7fac51ffb700 1 -- 172.21.15.103:0/4192539312 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fac44009cd0 con 0x7fac54133360 2023-07-02T09:04:27.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.767+0000 7fac51ffb700 1 -- 172.21.15.103:0/4192539312 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fac44009e30 con 0x7fac54133360 2023-07-02T09:04:27.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.767+0000 7fac51ffb700 1 -- 172.21.15.103:0/4192539312 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fac4400e360 con 0x7fac54133360 2023-07-02T09:04:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.768+0000 7fac59523700 1 -- 172.21.15.103:0/4192539312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54133360 msgr2=0x7fac5409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.768+0000 7fac59523700 1 --2- 172.21.15.103:0/4192539312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54133360 0x7fac5409cc90 crc :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.768+0000 7fac59523700 1 -- 172.21.15.103:0/4192539312 shutdown_connections 2023-07-02T09:04:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.768+0000 7fac59523700 1 --2- 172.21.15.103:0/4192539312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54133360 0x7fac5409cc90 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:27.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.768+0000 7fac59523700 1 -- 172.21.15.103:0/4192539312 >> 172.21.15.103:0/4192539312 conn(0x7fac540a2080 msgr2=0x7fac540a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.768+0000 7fac59523700 1 -- 172.21.15.103:0/4192539312 shutdown_connections 2023-07-02T09:04:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.768+0000 7fac59523700 1 -- 172.21.15.103:0/4192539312 wait complete. 2023-07-02T09:04:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.768+0000 7fac59523700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.769+0000 7fac59523700 1 Processor -- start 2023-07-02T09:04:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.769+0000 7fac59523700 1 -- start start 2023-07-02T09:04:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac59523700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54132f50 0x7fac541ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac59523700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5406d370 con 0x7fac54132f50 2023-07-02T09:04:27.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac52ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54132f50 0x7fac541ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:27.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac52ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54132f50 0x7fac541ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52506/0 (socket says 172.21.15.103:52506) 2023-07-02T09:04:27.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac52ffd700 1 -- 172.21.15.103:0/3160839971 learned_addr learned my addr 172.21.15.103:0/3160839971 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac52ffd700 1 -- 172.21.15.103:0/3160839971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5406dde0 con 0x7fac54132f50 2023-07-02T09:04:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac52ffd700 1 --2- 172.21.15.103:0/3160839971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54132f50 0x7fac541ba9b0 crc :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac3b7fe700 1 -- 172.21.15.103:0/3160839971 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fac48004b80 con 0x7fac54132f50 2023-07-02T09:04:27.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac3b7fe700 1 -- 172.21.15.103:0/3160839971 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fac48004ce0 con 0x7fac54132f50 2023-07-02T09:04:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac3b7fe700 1 -- 172.21.15.103:0/3160839971 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fac4800c4f0 con 0x7fac54132f50 2023-07-02T09:04:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac541ba4f0 con 0x7fac54132f50 2023-07-02T09:04:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.770+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac54071e40 con 0x7fac54132f50 2023-07-02T09:04:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.771+0000 7fac3b7fe700 1 -- 172.21.15.103:0/3160839971 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fac4800c650 con 0x7fac54132f50 2023-07-02T09:04:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.771+0000 7fac3b7fe700 1 --2- 172.21.15.103:0/3160839971 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac400377c0 0x7fac40039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.771+0000 7fac3b7fe700 1 -- 172.21.15.103:0/3160839971 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fac48046970 con 0x7fac54132f50 2023-07-02T09:04:27.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.772+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac54061ea0 con 0x7fac54132f50 2023-07-02T09:04:27.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.772+0000 7fac527fc700 1 --2- 172.21.15.103:0/3160839971 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac400377c0 0x7fac40039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:27.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.772+0000 7fac527fc700 1 --2- 172.21.15.103:0/3160839971 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac400377c0 0x7fac40039cd0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:27.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.776+0000 7fac3b7fe700 1 -- 172.21.15.103:0/3160839971 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fac48009c80 con 0x7fac54132f50 2023-07-02T09:04:27.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.973+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fac540725b0 con 0x7fac54132f50 2023-07-02T09:04:27.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.974+0000 7fac3b7fe700 1 -- 172.21.15.103:0/3160839971 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fac4802c020 con 0x7fac54132f50 2023-07-02T09:04:27.974 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:27.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac400377c0 msgr2=0x7fac40039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:27.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 --2- 172.21.15.103:0/3160839971 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac400377c0 0x7fac40039cd0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:27.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54132f50 msgr2=0x7fac541ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:27.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 --2- 172.21.15.103:0/3160839971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54132f50 0x7fac541ba9b0 crc :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:27.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 shutdown_connections 2023-07-02T09:04:27.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 --2- 172.21.15.103:0/3160839971 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fac400377c0 0x7fac40039cd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:27.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 --2- 172.21.15.103:0/3160839971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac54132f50 0x7fac541ba9b0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:27.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 >> 172.21.15.103:0/3160839971 conn(0x7fac540a2080 msgr2=0x7fac540a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:27.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 shutdown_connections 2023-07-02T09:04:27.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:27.976+0000 7fac59523700 1 -- 172.21.15.103:0/3160839971 wait complete. 2023-07-02T09:04:27.989 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:28.990 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:29.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.122+0000 7f04a2d9e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:29.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.128+0000 7f04a2d9e700 1 Processor -- start 2023-07-02T09:04:29.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.128+0000 7f04a2d9e700 1 -- start start 2023-07-02T09:04:29.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.128+0000 7f04a2d9e700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f049c133360 0x7f049c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:29.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.128+0000 7f04a2d9e700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c09e260 con 0x7f049c132f50 2023-07-02T09:04:29.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.128+0000 7f04a2d9e700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c0a1640 con 0x7f049c133360 2023-07-02T09:04:29.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f049c133360 0x7f049c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:29.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f04a0b3a700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f049c132f50 0x7f049c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46210/0 (socket says 172.21.15.103:46210) 2023-07-02T09:04:29.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049bfff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f049c133360 0x7f049c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52512/0 (socket says 172.21.15.103:52512) 2023-07-02T09:04:29.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f04a0b3a700 1 -- 172.21.15.103:0/3139053825 learned_addr learned my addr 172.21.15.103:0/3139053825 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:29.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049bfff700 1 -- 172.21.15.103:0/3139053825 >> v1:172.21.15.103:6789/0 conn(0x7f049c132f50 legacy=0x7f049c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:29.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049bfff700 1 -- 172.21.15.103:0/3139053825 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c132a20 con 0x7f049c133360 2023-07-02T09:04:29.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049b7fe700 1 -- 172.21.15.103:0/3139053825 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746443807 0 0) 0x7f049c09e260 con 0x7f049c132f50 2023-07-02T09:04:29.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049bfff700 1 --2- 172.21.15.103:0/3139053825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c133360 0x7f049c09cc90 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e598d6599407f75 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:29.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049b7fe700 1 -- 172.21.15.103:0/3139053825 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0488009cd0 con 0x7f049c133360 2023-07-02T09:04:29.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049b7fe700 1 -- 172.21.15.103:0/3139053825 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0488009e30 con 0x7f049c133360 2023-07-02T09:04:29.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.129+0000 7f049b7fe700 1 -- 172.21.15.103:0/3139053825 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f048800e360 con 0x7f049c133360 2023-07-02T09:04:29.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.130+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3139053825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c133360 msgr2=0x7f049c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:29.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.130+0000 7f04a2d9e700 1 --2- 172.21.15.103:0/3139053825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c133360 0x7f049c09cc90 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:29.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.130+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3139053825 shutdown_connections 2023-07-02T09:04:29.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.130+0000 7f04a2d9e700 1 --2- 172.21.15.103:0/3139053825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c133360 0x7f049c09cc90 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.130+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3139053825 >> 172.21.15.103:0/3139053825 conn(0x7f049c0a2080 msgr2=0x7f049c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.130+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3139053825 shutdown_connections 2023-07-02T09:04:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.130+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3139053825 wait complete. 2023-07-02T09:04:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.130+0000 7f04a2d9e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.131+0000 7f04a2d9e700 1 Processor -- start 2023-07-02T09:04:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.131+0000 7f04a2d9e700 1 -- start start 2023-07-02T09:04:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04a2d9e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c132f50 0x7f049c1ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04a2d9e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c06d370 con 0x7f049c132f50 2023-07-02T09:04:29.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04a0b3a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c132f50 0x7f049c1ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:29.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04a0b3a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c132f50 0x7f049c1ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52526/0 (socket says 172.21.15.103:52526) 2023-07-02T09:04:29.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04a0b3a700 1 -- 172.21.15.103:0/3205326829 learned_addr learned my addr 172.21.15.103:0/3205326829 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:29.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04a0b3a700 1 -- 172.21.15.103:0/3205326829 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c06dde0 con 0x7f049c132f50 2023-07-02T09:04:29.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04a0b3a700 1 --2- 172.21.15.103:0/3205326829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c132f50 0x7f049c1ba9b0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:29.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04997fa700 1 -- 172.21.15.103:0/3205326829 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f0490004b80 con 0x7f049c132f50 2023-07-02T09:04:29.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04997fa700 1 -- 172.21.15.103:0/3205326829 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f0490004ce0 con 0x7f049c132f50 2023-07-02T09:04:29.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c1ba4f0 con 0x7f049c132f50 2023-07-02T09:04:29.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.132+0000 7f04997fa700 1 -- 172.21.15.103:0/3205326829 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f049000c4f0 con 0x7f049c132f50 2023-07-02T09:04:29.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.133+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c071e40 con 0x7f049c132f50 2023-07-02T09:04:29.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.133+0000 7f04997fa700 1 -- 172.21.15.103:0/3205326829 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f049000c650 con 0x7f049c132f50 2023-07-02T09:04:29.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.134+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c061ea0 con 0x7f049c132f50 2023-07-02T09:04:29.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.134+0000 7f04997fa700 1 --2- 172.21.15.103:0/3205326829 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04800377c0 0x7f0480039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:29.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.134+0000 7f04997fa700 1 -- 172.21.15.103:0/3205326829 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f0490046970 con 0x7f049c132f50 2023-07-02T09:04:29.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.134+0000 7f049bfff700 1 --2- 172.21.15.103:0/3205326829 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04800377c0 0x7f0480039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:29.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.135+0000 7f049bfff700 1 --2- 172.21.15.103:0/3205326829 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04800377c0 0x7f0480039cd0 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.138+0000 7f04997fa700 1 -- 172.21.15.103:0/3205326829 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f0490009c80 con 0x7f049c132f50 2023-07-02T09:04:29.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.331+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f049c0725b0 con 0x7f049c132f50 2023-07-02T09:04:29.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.331+0000 7f04997fa700 1 -- 172.21.15.103:0/3205326829 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f04900098a0 con 0x7f049c132f50 2023-07-02T09:04:29.333 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:29.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04800377c0 msgr2=0x7f0480039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:29.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 --2- 172.21.15.103:0/3205326829 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04800377c0 0x7f0480039cd0 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:29.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c132f50 msgr2=0x7f049c1ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:29.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 --2- 172.21.15.103:0/3205326829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c132f50 0x7f049c1ba9b0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:29.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 shutdown_connections 2023-07-02T09:04:29.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 --2- 172.21.15.103:0/3205326829 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f04800377c0 0x7f0480039cd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:29.334 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 --2- 172.21.15.103:0/3205326829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c132f50 0x7f049c1ba9b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:29.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 >> 172.21.15.103:0/3205326829 conn(0x7f049c0a2080 msgr2=0x7f049c0a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:29.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 shutdown_connections 2023-07-02T09:04:29.335 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:29.333+0000 7f04a2d9e700 1 -- 172.21.15.103:0/3205326829 wait complete. 2023-07-02T09:04:29.346 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:30.348 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:30.492 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.488+0000 7fe959f45700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:30.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.493+0000 7fe959f45700 1 Processor -- start 2023-07-02T09:04:30.493 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.493+0000 7fe959f45700 1 -- start start 2023-07-02T09:04:30.494 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe959f45700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe95412f810 0x7fe95412fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:30.494 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe959f45700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95409f020 con 0x7fe95409d620 2023-07-02T09:04:30.494 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe959f45700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe954072670 con 0x7fe95412f810 2023-07-02T09:04:30.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe9537fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe95412f810 0x7fe95412fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:30.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe952ffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe95409d620 0x7fe95409da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46222/0 (socket says 172.21.15.103:46222) 2023-07-02T09:04:30.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe9537fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe95412f810 0x7fe95412fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52538/0 (socket says 172.21.15.103:52538) 2023-07-02T09:04:30.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe952ffd700 1 -- 172.21.15.103:0/4009657501 learned_addr learned my addr 172.21.15.103:0/4009657501 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:30.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe9537fe700 1 -- 172.21.15.103:0/4009657501 >> v1:172.21.15.103:6789/0 conn(0x7fe95409d620 legacy=0x7fe95409da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:30.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe9537fe700 1 -- 172.21.15.103:0/4009657501 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95409ab10 con 0x7fe95412f810 2023-07-02T09:04:30.495 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe9537fe700 1 --2- 172.21.15.103:0/4009657501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95412f810 0x7fe95412fc20 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463fafa06548f2e7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:30.496 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe9527fc700 1 -- 172.21.15.103:0/4009657501 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe93c009cd0 con 0x7fe95412f810 2023-07-02T09:04:30.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe9527fc700 1 -- 172.21.15.103:0/4009657501 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe93c009e30 con 0x7fe95412f810 2023-07-02T09:04:30.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.494+0000 7fe9527fc700 1 -- 172.21.15.103:0/4009657501 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe93c00e360 con 0x7fe95412f810 2023-07-02T09:04:30.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.495+0000 7fe959f45700 1 -- 172.21.15.103:0/4009657501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95412f810 msgr2=0x7fe95412fc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:30.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.495+0000 7fe959f45700 1 --2- 172.21.15.103:0/4009657501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95412f810 0x7fe95412fc20 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:30.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.495+0000 7fe959f45700 1 -- 172.21.15.103:0/4009657501 shutdown_connections 2023-07-02T09:04:30.497 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.495+0000 7fe959f45700 1 --2- 172.21.15.103:0/4009657501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95412f810 0x7fe95412fc20 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:30.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.495+0000 7fe959f45700 1 -- 172.21.15.103:0/4009657501 >> 172.21.15.103:0/4009657501 conn(0x7fe9540a0380 msgr2=0x7fe9540a2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:30.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.495+0000 7fe959f45700 1 -- 172.21.15.103:0/4009657501 shutdown_connections 2023-07-02T09:04:30.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.495+0000 7fe959f45700 1 -- 172.21.15.103:0/4009657501 wait complete. 2023-07-02T09:04:30.498 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.496+0000 7fe959f45700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:30.499 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.497+0000 7fe959f45700 1 Processor -- start 2023-07-02T09:04:30.499 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.497+0000 7fe959f45700 1 -- start start 2023-07-02T09:04:30.499 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe959f45700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95409d620 0x7fe95411b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:30.500 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe959f45700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95411e200 con 0x7fe95409d620 2023-07-02T09:04:30.500 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe9537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95409d620 0x7fe95411b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:30.500 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe9537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95409d620 0x7fe95411b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52552/0 (socket says 172.21.15.103:52552) 2023-07-02T09:04:30.500 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe9537fe700 1 -- 172.21.15.103:0/3437599347 learned_addr learned my addr 172.21.15.103:0/3437599347 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:30.500 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe9537fe700 1 -- 172.21.15.103:0/3437599347 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95411afb0 con 0x7fe95409d620 2023-07-02T09:04:30.500 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe9537fe700 1 --2- 172.21.15.103:0/3437599347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95409d620 0x7fe95411b590 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:30.500 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe94bfff700 1 -- 172.21.15.103:0/3437599347 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe93c0135b0 con 0x7fe95409d620 2023-07-02T09:04:30.501 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe94bfff700 1 -- 172.21.15.103:0/3437599347 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe93c013710 con 0x7fe95409d620 2023-07-02T09:04:30.501 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe94bfff700 1 -- 172.21.15.103:0/3437599347 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe93c0139b0 con 0x7fe95409d620 2023-07-02T09:04:30.501 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.498+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95411aca0 con 0x7fe95409d620 2023-07-02T09:04:30.501 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.499+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95411e780 con 0x7fe95409d620 2023-07-02T09:04:30.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.499+0000 7fe94bfff700 1 -- 172.21.15.103:0/3437599347 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fe93c013b60 con 0x7fe95409d620 2023-07-02T09:04:30.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.500+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe954068070 con 0x7fe95409d620 2023-07-02T09:04:30.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.500+0000 7fe94bfff700 1 --2- 172.21.15.103:0/3437599347 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9340377c0 0x7fe934039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:30.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.500+0000 7fe94bfff700 1 -- 172.21.15.103:0/3437599347 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fe93c048390 con 0x7fe95409d620 2023-07-02T09:04:30.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.500+0000 7fe952ffd700 1 --2- 172.21.15.103:0/3437599347 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9340377c0 0x7fe934039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:30.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.501+0000 7fe952ffd700 1 --2- 172.21.15.103:0/3437599347 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9340377c0 0x7fe934039cd0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:30.504 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.504+0000 7fe94bfff700 1 -- 172.21.15.103:0/3437599347 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fe93c00d1c0 con 0x7fe95409d620 2023-07-02T09:04:30.693 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.693+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe95409b600 con 0x7fe95409d620 2023-07-02T09:04:30.694 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.694+0000 7fe94bfff700 1 -- 172.21.15.103:0/3437599347 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fe93c011320 con 0x7fe95409d620 2023-07-02T09:04:30.694 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9340377c0 msgr2=0x7fe934039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:30.696 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 --2- 172.21.15.103:0/3437599347 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9340377c0 0x7fe934039cd0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95409d620 msgr2=0x7fe95411b590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 --2- 172.21.15.103:0/3437599347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95409d620 0x7fe95411b590 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 shutdown_connections 2023-07-02T09:04:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 --2- 172.21.15.103:0/3437599347 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fe9340377c0 0x7fe934039cd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 --2- 172.21.15.103:0/3437599347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95409d620 0x7fe95411b590 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 >> 172.21.15.103:0/3437599347 conn(0x7fe9540a0380 msgr2=0x7fe9540a1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:30.697 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 shutdown_connections 2023-07-02T09:04:30.698 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:30.696+0000 7fe959f45700 1 -- 172.21.15.103:0/3437599347 wait complete. 2023-07-02T09:04:30.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:31.711 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:31.855 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.851+0000 7fc9dcf50700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:31.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.856+0000 7fc9dcf50700 1 Processor -- start 2023-07-02T09:04:31.856 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.856+0000 7fc9dcf50700 1 -- start start 2023-07-02T09:04:31.857 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9dcf50700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9d8133360 0x7fc9d809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:31.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9dcf50700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d809e260 con 0x7fc9d8132f50 2023-07-02T09:04:31.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9dcf50700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d80a1640 con 0x7fc9d8133360 2023-07-02T09:04:31.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d5d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9d8133360 0x7fc9d809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:31.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d659c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc9d8132f50 0x7fc9d809c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46230/0 (socket says 172.21.15.103:46230) 2023-07-02T09:04:31.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d659c700 1 -- 172.21.15.103:0/793747183 learned_addr learned my addr 172.21.15.103:0/793747183 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:31.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d5d9b700 1 -- 172.21.15.103:0/793747183 >> v1:172.21.15.103:6789/0 conn(0x7fc9d8132f50 legacy=0x7fc9d809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:31.858 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d5d9b700 1 -- 172.21.15.103:0/793747183 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d8132a20 con 0x7fc9d8133360 2023-07-02T09:04:31.859 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d559a700 1 -- 172.21.15.103:0/793747183 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383873350 0 0) 0x7fc9d809e260 con 0x7fc9d8132f50 2023-07-02T09:04:31.859 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d5d9b700 1 --2- 172.21.15.103:0/793747183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8133360 0x7fc9d809cc90 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b63de9445c61b3d server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:31.859 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d559a700 1 -- 172.21.15.103:0/793747183 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9cc009cd0 con 0x7fc9d8133360 2023-07-02T09:04:31.859 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d559a700 1 -- 172.21.15.103:0/793747183 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc9cc009e30 con 0x7fc9d8133360 2023-07-02T09:04:31.859 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.857+0000 7fc9d559a700 1 -- 172.21.15.103:0/793747183 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9cc00e360 con 0x7fc9d8133360 2023-07-02T09:04:31.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.858+0000 7fc9dcf50700 1 -- 172.21.15.103:0/793747183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8133360 msgr2=0x7fc9d809cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:31.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.858+0000 7fc9dcf50700 1 --2- 172.21.15.103:0/793747183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8133360 0x7fc9d809cc90 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:31.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.858+0000 7fc9dcf50700 1 -- 172.21.15.103:0/793747183 shutdown_connections 2023-07-02T09:04:31.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.858+0000 7fc9dcf50700 1 --2- 172.21.15.103:0/793747183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8133360 0x7fc9d809cc90 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:31.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.858+0000 7fc9dcf50700 1 -- 172.21.15.103:0/793747183 >> 172.21.15.103:0/793747183 conn(0x7fc9d80a2080 msgr2=0x7fc9d80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.858+0000 7fc9dcf50700 1 -- 172.21.15.103:0/793747183 shutdown_connections 2023-07-02T09:04:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.858+0000 7fc9dcf50700 1 -- 172.21.15.103:0/793747183 wait complete. 2023-07-02T09:04:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.858+0000 7fc9dcf50700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.859+0000 7fc9dcf50700 1 Processor -- start 2023-07-02T09:04:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.859+0000 7fc9dcf50700 1 -- start start 2023-07-02T09:04:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.860+0000 7fc9dcf50700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8132f50 0x7fc9d812c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.860+0000 7fc9dcf50700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d806d2f0 con 0x7fc9d8132f50 2023-07-02T09:04:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.860+0000 7fc9d659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8132f50 0x7fc9d812c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.860+0000 7fc9d659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8132f50 0x7fc9d812c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52570/0 (socket says 172.21.15.103:52570) 2023-07-02T09:04:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.860+0000 7fc9d659c700 1 -- 172.21.15.103:0/486974869 learned_addr learned my addr 172.21.15.103:0/486974869 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.860+0000 7fc9d659c700 1 -- 172.21.15.103:0/486974869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d806dd60 con 0x7fc9d8132f50 2023-07-02T09:04:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.861+0000 7fc9d659c700 1 --2- 172.21.15.103:0/486974869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8132f50 0x7fc9d812c1a0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.861+0000 7fc9c6ffd700 1 -- 172.21.15.103:0/486974869 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9c8004b80 con 0x7fc9d8132f50 2023-07-02T09:04:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.861+0000 7fc9c6ffd700 1 -- 172.21.15.103:0/486974869 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc9c8004ce0 con 0x7fc9d8132f50 2023-07-02T09:04:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.861+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d812bc50 con 0x7fc9d8132f50 2023-07-02T09:04:31.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.861+0000 7fc9c6ffd700 1 -- 172.21.15.103:0/486974869 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9c800c4f0 con 0x7fc9d8132f50 2023-07-02T09:04:31.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.861+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d806f850 con 0x7fc9d8132f50 2023-07-02T09:04:31.865 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.862+0000 7fc9c6ffd700 1 -- 172.21.15.103:0/486974869 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fc9c800c650 con 0x7fc9d8132f50 2023-07-02T09:04:31.865 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.862+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d806dbb0 con 0x7fc9d8132f50 2023-07-02T09:04:31.865 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.862+0000 7fc9c6ffd700 1 --2- 172.21.15.103:0/486974869 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc9b80377c0 0x7fc9b8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:31.865 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.862+0000 7fc9c6ffd700 1 -- 172.21.15.103:0/486974869 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fc9c8046970 con 0x7fc9d8132f50 2023-07-02T09:04:31.865 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.862+0000 7fc9d5d9b700 1 --2- 172.21.15.103:0/486974869 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc9b80377c0 0x7fc9b8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:31.865 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.863+0000 7fc9d5d9b700 1 --2- 172.21.15.103:0/486974869 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc9b80377c0 0x7fc9b8039cd0 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:31.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:31.867+0000 7fc9c6ffd700 1 -- 172.21.15.103:0/486974869 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fc9c8009c80 con 0x7fc9d8132f50 2023-07-02T09:04:32.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.059+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc9d8061ea0 con 0x7fc9d8132f50 2023-07-02T09:04:32.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.059+0000 7fc9c6ffd700 1 -- 172.21.15.103:0/486974869 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fc9c80098a0 con 0x7fc9d8132f50 2023-07-02T09:04:32.059 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:32.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc9b80377c0 msgr2=0x7fc9b8039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:32.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 --2- 172.21.15.103:0/486974869 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc9b80377c0 0x7fc9b8039cd0 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:32.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8132f50 msgr2=0x7fc9d812c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:32.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 --2- 172.21.15.103:0/486974869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8132f50 0x7fc9d812c1a0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:32.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 shutdown_connections 2023-07-02T09:04:32.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 --2- 172.21.15.103:0/486974869 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fc9b80377c0 0x7fc9b8039cd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:32.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 --2- 172.21.15.103:0/486974869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d8132f50 0x7fc9d812c1a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:32.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 >> 172.21.15.103:0/486974869 conn(0x7fc9d80a2080 msgr2=0x7fc9d8130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:32.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 shutdown_connections 2023-07-02T09:04:32.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:32.061+0000 7fc9dcf50700 1 -- 172.21.15.103:0/486974869 wait complete. 2023-07-02T09:04:32.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~286s 2023-07-02T09:04:32.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:33.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:33.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.211+0000 7fa2f439c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:33.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.215+0000 7fa2f439c700 1 Processor -- start 2023-07-02T09:04:33.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.215+0000 7fa2f439c700 1 -- start start 2023-07-02T09:04:33.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.216+0000 7fa2f439c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa2ec12f250 0x7fa2ec12f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:33.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.216+0000 7fa2f439c700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ec06df60 con 0x7fa2ec098a00 2023-07-02T09:04:33.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.216+0000 7fa2f439c700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ec09a710 con 0x7fa2ec12f250 2023-07-02T09:04:33.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f2138700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa2ec12f250 0x7fa2ec12f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:33.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f1937700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa2ec098a00 0x7fa2ec098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:46236/0 (socket says 172.21.15.103:46236) 2023-07-02T09:04:33.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f1937700 1 -- 172.21.15.103:0/2932782915 learned_addr learned my addr 172.21.15.103:0/2932782915 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f2138700 1 -- 172.21.15.103:0/2932782915 >> v1:172.21.15.103:6789/0 conn(0x7fa2ec098a00 legacy=0x7fa2ec098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f1136700 1 -- 172.21.15.103:0/2932782915 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058187927 0 0) 0x7fa2ec06df60 con 0x7fa2ec098a00 2023-07-02T09:04:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f2138700 1 -- 172.21.15.103:0/2932782915 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec12ed20 con 0x7fa2ec12f250 2023-07-02T09:04:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f2138700 1 --2- 172.21.15.103:0/2932782915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec12f250 0x7fa2ec12f660 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a86729677807fdd server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f1136700 1 -- 172.21.15.103:0/2932782915 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa2dc00ac60 con 0x7fa2ec12f250 2023-07-02T09:04:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f1136700 1 -- 172.21.15.103:0/2932782915 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa2dc00adc0 con 0x7fa2ec12f250 2023-07-02T09:04:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.217+0000 7fa2f1136700 1 -- 172.21.15.103:0/2932782915 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa2dc00f470 con 0x7fa2ec12f250 2023-07-02T09:04:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.218+0000 7fa2f439c700 1 -- 172.21.15.103:0/2932782915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec12f250 msgr2=0x7fa2ec12f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.218+0000 7fa2f439c700 1 --2- 172.21.15.103:0/2932782915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec12f250 0x7fa2ec12f660 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.218+0000 7fa2f439c700 1 -- 172.21.15.103:0/2932782915 shutdown_connections 2023-07-02T09:04:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.218+0000 7fa2f439c700 1 --2- 172.21.15.103:0/2932782915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec12f250 0x7fa2ec12f660 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.218+0000 7fa2f439c700 1 -- 172.21.15.103:0/2932782915 >> 172.21.15.103:0/2932782915 conn(0x7fa2ec09d810 msgr2=0x7fa2ec09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.218+0000 7fa2f439c700 1 -- 172.21.15.103:0/2932782915 shutdown_connections 2023-07-02T09:04:33.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.218+0000 7fa2f439c700 1 -- 172.21.15.103:0/2932782915 wait complete. 2023-07-02T09:04:33.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.218+0000 7fa2f439c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:33.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.219+0000 7fa2f439c700 1 Processor -- start 2023-07-02T09:04:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.219+0000 7fa2f439c700 1 -- start start 2023-07-02T09:04:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f439c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec098a00 0x7fa2ec117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f439c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec11a390 con 0x7fa2ec098a00 2023-07-02T09:04:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f2138700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec098a00 0x7fa2ec117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f2138700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec098a00 0x7fa2ec117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52578/0 (socket says 172.21.15.103:52578) 2023-07-02T09:04:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f2138700 1 -- 172.21.15.103:0/486375694 learned_addr learned my addr 172.21.15.103:0/486375694 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:33.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f2138700 1 -- 172.21.15.103:0/486375694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec099b60 con 0x7fa2ec098a00 2023-07-02T09:04:33.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f2138700 1 --2- 172.21.15.103:0/486375694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec098a00 0x7fa2ec117360 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:33.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2e2ffd700 1 -- 172.21.15.103:0/486375694 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa2dc00f670 con 0x7fa2ec098a00 2023-07-02T09:04:33.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2e2ffd700 1 -- 172.21.15.103:0/486375694 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa2dc00f7d0 con 0x7fa2ec098a00 2023-07-02T09:04:33.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ec1196f0 con 0x7fa2ec098a00 2023-07-02T09:04:33.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2e2ffd700 1 -- 172.21.15.103:0/486375694 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa2dc00fa70 con 0x7fa2ec098a00 2023-07-02T09:04:33.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.220+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ec119db0 con 0x7fa2ec098a00 2023-07-02T09:04:33.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.221+0000 7fa2e2ffd700 1 -- 172.21.15.103:0/486375694 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa2dc00fc20 con 0x7fa2ec098a00 2023-07-02T09:04:33.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.221+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ec061ea0 con 0x7fa2ec098a00 2023-07-02T09:04:33.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.221+0000 7fa2e2ffd700 1 --2- 172.21.15.103:0/486375694 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2d00377c0 0x7fa2d0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:33.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.221+0000 7fa2e2ffd700 1 -- 172.21.15.103:0/486375694 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa2dc049df0 con 0x7fa2ec098a00 2023-07-02T09:04:33.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.222+0000 7fa2f1937700 1 --2- 172.21.15.103:0/486375694 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2d00377c0 0x7fa2d0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:33.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.222+0000 7fa2f1937700 1 --2- 172.21.15.103:0/486375694 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2d00377c0 0x7fa2d0039cd0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:33.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.226+0000 7fa2e2ffd700 1 -- 172.21.15.103:0/486375694 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa2dc0103b0 con 0x7fa2ec098a00 2023-07-02T09:04:33.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.418+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa2ec099b60 con 0x7fa2ec098a00 2023-07-02T09:04:33.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.419+0000 7fa2e2ffd700 1 -- 172.21.15.103:0/486375694 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa2ec099b60 con 0x7fa2ec098a00 2023-07-02T09:04:33.419 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:33.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2d00377c0 msgr2=0x7fa2d0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:33.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 --2- 172.21.15.103:0/486375694 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2d00377c0 0x7fa2d0039cd0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:33.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec098a00 msgr2=0x7fa2ec117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:33.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 --2- 172.21.15.103:0/486375694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec098a00 0x7fa2ec117360 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:33.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 shutdown_connections 2023-07-02T09:04:33.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 --2- 172.21.15.103:0/486375694 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa2d00377c0 0x7fa2d0039cd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:33.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 --2- 172.21.15.103:0/486375694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ec098a00 0x7fa2ec117360 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:33.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 >> 172.21.15.103:0/486375694 conn(0x7fa2ec09d810 msgr2=0x7fa2ec0a0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:33.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 shutdown_connections 2023-07-02T09:04:33.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:33.421+0000 7fa2f439c700 1 -- 172.21.15.103:0/486375694 wait complete. 2023-07-02T09:04:33.434 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:34.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:34.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.574+0000 7f14c9a3a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:34.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.579+0000 7f14c9a3a700 1 Processor -- start 2023-07-02T09:04:34.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.580+0000 7f14c9a3a700 1 -- start start 2023-07-02T09:04:34.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.580+0000 7f14c9a3a700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14c4132f50 0x7f14c4133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:34.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.580+0000 7f14c9a3a700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c409e260 con 0x7f14c409c580 2023-07-02T09:04:34.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.580+0000 7f14c9a3a700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c40a1640 con 0x7f14c4132f50 2023-07-02T09:04:34.581 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.580+0000 7f14c2ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14c4132f50 0x7f14c4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:34.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.580+0000 7f14c27fc700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f14c409c580 0x7f14c409c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50018/0 (socket says 172.21.15.103:50018) 2023-07-02T09:04:34.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.580+0000 7f14c2ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f14c4132f50 0x7f14c4133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56418/0 (socket says 172.21.15.103:56418) 2023-07-02T09:04:34.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.580+0000 7f14c27fc700 1 -- 172.21.15.103:0/3668758520 learned_addr learned my addr 172.21.15.103:0/3668758520 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:34.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.581+0000 7f14c2ffd700 1 -- 172.21.15.103:0/3668758520 >> v1:172.21.15.103:6789/0 conn(0x7f14c409c580 legacy=0x7f14c409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:34.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.581+0000 7f14c2ffd700 1 -- 172.21.15.103:0/3668758520 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c4132a20 con 0x7f14c4132f50 2023-07-02T09:04:34.582 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.581+0000 7f14c1ffb700 1 -- 172.21.15.103:0/3668758520 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779064152 0 0) 0x7f14c409e260 con 0x7f14c409c580 2023-07-02T09:04:34.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.581+0000 7f14c2ffd700 1 --2- 172.21.15.103:0/3668758520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4132f50 0x7f14c4133360 crc :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8c546defc70d82 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:34.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.581+0000 7f14c1ffb700 1 -- 172.21.15.103:0/3668758520 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f14b8009cd0 con 0x7f14c4132f50 2023-07-02T09:04:34.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.581+0000 7f14c1ffb700 1 -- 172.21.15.103:0/3668758520 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f14b8009e30 con 0x7f14c4132f50 2023-07-02T09:04:34.583 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.581+0000 7f14c1ffb700 1 -- 172.21.15.103:0/3668758520 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f14b800e360 con 0x7f14c4132f50 2023-07-02T09:04:34.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.582+0000 7f14c9a3a700 1 -- 172.21.15.103:0/3668758520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4132f50 msgr2=0x7f14c4133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:34.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.582+0000 7f14c9a3a700 1 --2- 172.21.15.103:0/3668758520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4132f50 0x7f14c4133360 crc :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:34.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.582+0000 7f14c9a3a700 1 -- 172.21.15.103:0/3668758520 shutdown_connections 2023-07-02T09:04:34.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.582+0000 7f14c9a3a700 1 --2- 172.21.15.103:0/3668758520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c4132f50 0x7f14c4133360 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:34.585 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.582+0000 7f14c9a3a700 1 -- 172.21.15.103:0/3668758520 >> 172.21.15.103:0/3668758520 conn(0x7f14c40a2080 msgr2=0x7f14c40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:34.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.582+0000 7f14c9a3a700 1 -- 172.21.15.103:0/3668758520 shutdown_connections 2023-07-02T09:04:34.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.582+0000 7f14c9a3a700 1 -- 172.21.15.103:0/3668758520 wait complete. 2023-07-02T09:04:34.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.582+0000 7f14c9a3a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:34.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.583+0000 7f14c9a3a700 1 Processor -- start 2023-07-02T09:04:34.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.583+0000 7f14c9a3a700 1 -- start start 2023-07-02T09:04:34.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c9a3a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c409c580 0x7f14c41baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:34.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c9a3a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c41ba610 con 0x7f14c409c580 2023-07-02T09:04:34.586 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c409c580 0x7f14c41baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:34.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c409c580 0x7f14c41baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56432/0 (socket says 172.21.15.103:56432) 2023-07-02T09:04:34.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c2ffd700 1 -- 172.21.15.103:0/485961424 learned_addr learned my addr 172.21.15.103:0/485961424 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:34.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c2ffd700 1 -- 172.21.15.103:0/485961424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c406dfc0 con 0x7f14c409c580 2023-07-02T09:04:34.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c2ffd700 1 --2- 172.21.15.103:0/485961424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c409c580 0x7f14c41baa60 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:34.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14abfff700 1 -- 172.21.15.103:0/485961424 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f14b8013530 con 0x7f14c409c580 2023-07-02T09:04:34.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14abfff700 1 -- 172.21.15.103:0/485961424 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f14b8013690 con 0x7f14c409c580 2023-07-02T09:04:34.587 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14abfff700 1 -- 172.21.15.103:0/485961424 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f14b8013930 con 0x7f14c409c580 2023-07-02T09:04:34.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c41ba270 con 0x7f14c409c580 2023-07-02T09:04:34.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.584+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c406dba0 con 0x7f14c409c580 2023-07-02T09:04:34.588 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.585+0000 7f14abfff700 1 -- 172.21.15.103:0/485961424 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f14b8013ae0 con 0x7f14c409c580 2023-07-02T09:04:34.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.585+0000 7f14abfff700 1 --2- 172.21.15.103:0/485961424 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377b0 0x7f14b0039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:34.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.585+0000 7f14abfff700 1 -- 172.21.15.103:0/485961424 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f14b8018040 con 0x7f14c409c580 2023-07-02T09:04:34.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.585+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c41ba460 con 0x7f14c409c580 2023-07-02T09:04:34.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.586+0000 7f14c27fc700 1 --2- 172.21.15.103:0/485961424 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377b0 0x7f14b0039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:34.589 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.586+0000 7f14c27fc700 1 --2- 172.21.15.103:0/485961424 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377b0 0x7f14b0039cc0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:34.590 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.590+0000 7f14abfff700 1 -- 172.21.15.103:0/485961424 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f14b8018e00 con 0x7f14c409c580 2023-07-02T09:04:34.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.779+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f14c4061ea0 con 0x7f14c409c580 2023-07-02T09:04:34.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.779+0000 7f14abfff700 1 -- 172.21.15.103:0/485961424 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f14b8010e60 con 0x7f14c409c580 2023-07-02T09:04:34.780 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:34.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377b0 msgr2=0x7f14b0039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:34.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 --2- 172.21.15.103:0/485961424 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377b0 0x7f14b0039cc0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:34.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c409c580 msgr2=0x7f14c41baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:34.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 --2- 172.21.15.103:0/485961424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c409c580 0x7f14c41baa60 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:34.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 shutdown_connections 2023-07-02T09:04:34.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 --2- 172.21.15.103:0/485961424 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f14b00377b0 0x7f14b0039cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:34.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 --2- 172.21.15.103:0/485961424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14c409c580 0x7f14c41baa60 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:34.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 >> 172.21.15.103:0/485961424 conn(0x7f14c40a2080 msgr2=0x7f14c4130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:34.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 shutdown_connections 2023-07-02T09:04:34.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:34.781+0000 7f14c9a3a700 1 -- 172.21.15.103:0/485961424 wait complete. 2023-07-02T09:04:34.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:35.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:35.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.936+0000 7fa62db24700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:35.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.941+0000 7fa62db24700 1 Processor -- start 2023-07-02T09:04:35.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.941+0000 7fa62db24700 1 -- start start 2023-07-02T09:04:35.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa62db24700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa628132030 0x7fa628134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:35.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa62db24700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62806df60 con 0x7fa62812f250 2023-07-02T09:04:35.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa62db24700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62809a710 con 0x7fa628132030 2023-07-02T09:04:35.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa626ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa628132030 0x7fa628134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:35.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa6277fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa62812f250 0x7fa62812f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50026/0 (socket says 172.21.15.103:50026) 2023-07-02T09:04:35.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa626ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa628132030 0x7fa628134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56438/0 (socket says 172.21.15.103:56438) 2023-07-02T09:04:35.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa6277fe700 1 -- 172.21.15.103:0/1766644908 learned_addr learned my addr 172.21.15.103:0/1766644908 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:35.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa626ffd700 1 -- 172.21.15.103:0/1766644908 >> v1:172.21.15.103:6789/0 conn(0x7fa62812f250 legacy=0x7fa62812f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:35.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa626ffd700 1 -- 172.21.15.103:0/1766644908 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62812ed20 con 0x7fa628132030 2023-07-02T09:04:35.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa6267fc700 1 -- 172.21.15.103:0/1766644908 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232007102 0 0) 0x7fa62806df60 con 0x7fa62812f250 2023-07-02T09:04:35.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa626ffd700 1 --2- 172.21.15.103:0/1766644908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa628132030 0x7fa628134450 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acad09ea7631f1a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:35.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa6267fc700 1 -- 172.21.15.103:0/1766644908 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa610009cd0 con 0x7fa628132030 2023-07-02T09:04:35.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa6267fc700 1 -- 172.21.15.103:0/1766644908 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa610009e30 con 0x7fa628132030 2023-07-02T09:04:35.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.942+0000 7fa6267fc700 1 -- 172.21.15.103:0/1766644908 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa61000e360 con 0x7fa628132030 2023-07-02T09:04:35.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.943+0000 7fa62db24700 1 -- 172.21.15.103:0/1766644908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa628132030 msgr2=0x7fa628134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:35.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.943+0000 7fa62db24700 1 --2- 172.21.15.103:0/1766644908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa628132030 0x7fa628134450 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:35.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.943+0000 7fa62db24700 1 -- 172.21.15.103:0/1766644908 shutdown_connections 2023-07-02T09:04:35.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.943+0000 7fa62db24700 1 --2- 172.21.15.103:0/1766644908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa628132030 0x7fa628134450 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:35.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.943+0000 7fa62db24700 1 -- 172.21.15.103:0/1766644908 >> 172.21.15.103:0/1766644908 conn(0x7fa62809d810 msgr2=0x7fa62809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:35.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.943+0000 7fa62db24700 1 -- 172.21.15.103:0/1766644908 shutdown_connections 2023-07-02T09:04:35.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.943+0000 7fa62db24700 1 -- 172.21.15.103:0/1766644908 wait complete. 2023-07-02T09:04:35.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.943+0000 7fa62db24700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:35.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.944+0000 7fa62db24700 1 Processor -- start 2023-07-02T09:04:35.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.944+0000 7fa62db24700 1 -- start start 2023-07-02T09:04:35.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.945+0000 7fa62db24700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62812f250 0x7fa628118110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.945+0000 7fa62db24700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6280705a0 con 0x7fa62812f250 2023-07-02T09:04:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.945+0000 7fa6277fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62812f250 0x7fa628118110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.945+0000 7fa6277fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62812f250 0x7fa628118110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56448/0 (socket says 172.21.15.103:56448) 2023-07-02T09:04:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.945+0000 7fa6277fe700 1 -- 172.21.15.103:0/3391896749 learned_addr learned my addr 172.21.15.103:0/3391896749 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.945+0000 7fa6277fe700 1 -- 172.21.15.103:0/3391896749 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6281178a0 con 0x7fa62812f250 2023-07-02T09:04:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.946+0000 7fa6277fe700 1 --2- 172.21.15.103:0/3391896749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62812f250 0x7fa628118110 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.946+0000 7fa61ffff700 1 -- 172.21.15.103:0/3391896749 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa618004bc0 con 0x7fa62812f250 2023-07-02T09:04:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.946+0000 7fa61ffff700 1 -- 172.21.15.103:0/3391896749 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa618004d20 con 0x7fa62812f250 2023-07-02T09:04:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.946+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa628117360 con 0x7fa62812f250 2023-07-02T09:04:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.946+0000 7fa61ffff700 1 -- 172.21.15.103:0/3391896749 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa61800d150 con 0x7fa62812f250 2023-07-02T09:04:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.946+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa628071a70 con 0x7fa62812f250 2023-07-02T09:04:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.947+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa628117710 con 0x7fa62812f250 2023-07-02T09:04:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.947+0000 7fa61ffff700 1 -- 172.21.15.103:0/3391896749 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fa61800d2b0 con 0x7fa62812f250 2023-07-02T09:04:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.947+0000 7fa61ffff700 1 --2- 172.21.15.103:0/3391896749 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6080377c0 0x7fa608039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.947+0000 7fa61ffff700 1 -- 172.21.15.103:0/3391896749 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fa618047100 con 0x7fa62812f250 2023-07-02T09:04:35.951 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.947+0000 7fa626ffd700 1 --2- 172.21.15.103:0/3391896749 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6080377c0 0x7fa608039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:35.951 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.948+0000 7fa626ffd700 1 --2- 172.21.15.103:0/3391896749 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6080377c0 0x7fa608039cd0 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:35.951 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:35.952+0000 7fa61ffff700 1 -- 172.21.15.103:0/3391896749 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fa618005870 con 0x7fa62812f250 2023-07-02T09:04:36.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.142+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa628061ea0 con 0x7fa62812f250 2023-07-02T09:04:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.143+0000 7fa61ffff700 1 -- 172.21.15.103:0/3391896749 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fa618005040 con 0x7fa62812f250 2023-07-02T09:04:36.143 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:36.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6080377c0 msgr2=0x7fa608039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:36.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 --2- 172.21.15.103:0/3391896749 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6080377c0 0x7fa608039cd0 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:36.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62812f250 msgr2=0x7fa628118110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:36.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 --2- 172.21.15.103:0/3391896749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62812f250 0x7fa628118110 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:36.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 shutdown_connections 2023-07-02T09:04:36.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 --2- 172.21.15.103:0/3391896749 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fa6080377c0 0x7fa608039cd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:36.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 --2- 172.21.15.103:0/3391896749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62812f250 0x7fa628118110 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:36.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 >> 172.21.15.103:0/3391896749 conn(0x7fa62809d810 msgr2=0x7fa62812e070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:36.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 shutdown_connections 2023-07-02T09:04:36.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:36.145+0000 7fa62db24700 1 -- 172.21.15.103:0/3391896749 wait complete. 2023-07-02T09:04:36.158 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:37.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:37.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.299+0000 7fb7dca45700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:37.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.304+0000 7fb7dca45700 1 Processor -- start 2023-07-02T09:04:37.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.304+0000 7fb7dca45700 1 -- start start 2023-07-02T09:04:37.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7dca45700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb7d8132f50 0x7fb7d8133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:37.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7dca45700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d809e260 con 0x7fb7d809c580 2023-07-02T09:04:37.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7dca45700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d80a1640 con 0x7fb7d8132f50 2023-07-02T09:04:37.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7d659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb7d8132f50 0x7fb7d8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:37.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7d5d9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb7d809c580 0x7fb7d809c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50032/0 (socket says 172.21.15.103:50032) 2023-07-02T09:04:37.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7d659c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb7d8132f50 0x7fb7d8133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56452/0 (socket says 172.21.15.103:56452) 2023-07-02T09:04:37.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7d5d9b700 1 -- 172.21.15.103:0/1276910032 learned_addr learned my addr 172.21.15.103:0/1276910032 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:37.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7d659c700 1 -- 172.21.15.103:0/1276910032 >> v1:172.21.15.103:6789/0 conn(0x7fb7d809c580 legacy=0x7fb7d809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:37.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.306+0000 7fb7d659c700 1 -- 172.21.15.103:0/1276910032 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d8132a20 con 0x7fb7d8132f50 2023-07-02T09:04:37.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.305+0000 7fb7d559a700 1 -- 172.21.15.103:0/1276910032 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854136730 0 0) 0x7fb7d809e260 con 0x7fb7d809c580 2023-07-02T09:04:37.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.306+0000 7fb7d659c700 1 --2- 172.21.15.103:0/1276910032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d8132f50 0x7fb7d8133360 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1001658d591f57 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:37.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.306+0000 7fb7d559a700 1 -- 172.21.15.103:0/1276910032 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb7cc009cd0 con 0x7fb7d8132f50 2023-07-02T09:04:37.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.306+0000 7fb7d559a700 1 -- 172.21.15.103:0/1276910032 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb7cc009e30 con 0x7fb7d8132f50 2023-07-02T09:04:37.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.306+0000 7fb7d559a700 1 -- 172.21.15.103:0/1276910032 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb7cc00e360 con 0x7fb7d8132f50 2023-07-02T09:04:37.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.306+0000 7fb7dca45700 1 -- 172.21.15.103:0/1276910032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d8132f50 msgr2=0x7fb7d8133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:37.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.306+0000 7fb7dca45700 1 --2- 172.21.15.103:0/1276910032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d8132f50 0x7fb7d8133360 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:37.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.307+0000 7fb7dca45700 1 -- 172.21.15.103:0/1276910032 shutdown_connections 2023-07-02T09:04:37.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.307+0000 7fb7dca45700 1 --2- 172.21.15.103:0/1276910032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d8132f50 0x7fb7d8133360 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:37.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.307+0000 7fb7dca45700 1 -- 172.21.15.103:0/1276910032 >> 172.21.15.103:0/1276910032 conn(0x7fb7d80a2080 msgr2=0x7fb7d80a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:37.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.307+0000 7fb7dca45700 1 -- 172.21.15.103:0/1276910032 shutdown_connections 2023-07-02T09:04:37.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.307+0000 7fb7dca45700 1 -- 172.21.15.103:0/1276910032 wait complete. 2023-07-02T09:04:37.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.307+0000 7fb7dca45700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:37.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.308+0000 7fb7dca45700 1 Processor -- start 2023-07-02T09:04:37.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.308+0000 7fb7dca45700 1 -- start start 2023-07-02T09:04:37.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7dca45700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d809c580 0x7fb7d81baa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:37.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7dca45700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81ba610 con 0x7fb7d809c580 2023-07-02T09:04:37.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7d659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d809c580 0x7fb7d81baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:37.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7d659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d809c580 0x7fb7d81baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56466/0 (socket says 172.21.15.103:56466) 2023-07-02T09:04:37.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7d659c700 1 -- 172.21.15.103:0/4243258417 learned_addr learned my addr 172.21.15.103:0/4243258417 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:37.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7d659c700 1 -- 172.21.15.103:0/4243258417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d806dfc0 con 0x7fb7d809c580 2023-07-02T09:04:37.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7d659c700 1 --2- 172.21.15.103:0/4243258417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d809c580 0x7fb7d81baa60 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:37.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7c6ffd700 1 -- 172.21.15.103:0/4243258417 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb7cc013530 con 0x7fb7d809c580 2023-07-02T09:04:37.313 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7c6ffd700 1 -- 172.21.15.103:0/4243258417 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb7cc013690 con 0x7fb7d809c580 2023-07-02T09:04:37.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d81ba270 con 0x7fb7d809c580 2023-07-02T09:04:37.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7c6ffd700 1 -- 172.21.15.103:0/4243258417 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb7cc013930 con 0x7fb7d809c580 2023-07-02T09:04:37.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.309+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d806dba0 con 0x7fb7d809c580 2023-07-02T09:04:37.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.310+0000 7fb7c6ffd700 1 -- 172.21.15.103:0/4243258417 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb7cc013ae0 con 0x7fb7d809c580 2023-07-02T09:04:37.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.310+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d81ba460 con 0x7fb7d809c580 2023-07-02T09:04:37.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.310+0000 7fb7c6ffd700 1 --2- 172.21.15.103:0/4243258417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7b80377b0 0x7fb7b8039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:37.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.310+0000 7fb7c6ffd700 1 -- 172.21.15.103:0/4243258417 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb7cc018040 con 0x7fb7d809c580 2023-07-02T09:04:37.314 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.311+0000 7fb7d5d9b700 1 --2- 172.21.15.103:0/4243258417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7b80377b0 0x7fb7b8039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:37.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.311+0000 7fb7d5d9b700 1 --2- 172.21.15.103:0/4243258417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7b80377b0 0x7fb7b8039cc0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:37.315 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.315+0000 7fb7c6ffd700 1 -- 172.21.15.103:0/4243258417 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb7cc018e00 con 0x7fb7d809c580 2023-07-02T09:04:37.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.505+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb7d8061ea0 con 0x7fb7d809c580 2023-07-02T09:04:37.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.505+0000 7fb7c6ffd700 1 -- 172.21.15.103:0/4243258417 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb7cc010e60 con 0x7fb7d809c580 2023-07-02T09:04:37.506 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:37.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7b80377b0 msgr2=0x7fb7b8039cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:37.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 --2- 172.21.15.103:0/4243258417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7b80377b0 0x7fb7b8039cc0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:37.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d809c580 msgr2=0x7fb7d81baa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:37.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 --2- 172.21.15.103:0/4243258417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d809c580 0x7fb7d81baa60 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:37.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 shutdown_connections 2023-07-02T09:04:37.508 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 --2- 172.21.15.103:0/4243258417 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb7b80377b0 0x7fb7b8039cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:37.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 --2- 172.21.15.103:0/4243258417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7d809c580 0x7fb7d81baa60 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:37.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 >> 172.21.15.103:0/4243258417 conn(0x7fb7d80a2080 msgr2=0x7fb7d8130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:37.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 shutdown_connections 2023-07-02T09:04:37.509 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:37.507+0000 7fb7dca45700 1 -- 172.21.15.103:0/4243258417 wait complete. 2023-07-02T09:04:37.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:37.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~292s 2023-07-02T09:04:38.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:38.666 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.661+0000 7f97a11c1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:38.667 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.666+0000 7f97a11c1700 1 Processor -- start 2023-07-02T09:04:38.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.666+0000 7f97a11c1700 1 -- start start 2023-07-02T09:04:38.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f97a11c1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f979c133360 0x7f979c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:38.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f97a11c1700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c09e260 con 0x7f979c132f50 2023-07-02T09:04:38.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f97a11c1700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979c0a1640 con 0x7f979c133360 2023-07-02T09:04:38.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f979a59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f979c133360 0x7f979c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:38.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f979ad9d700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f979c132f50 0x7f979c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50038/0 (socket says 172.21.15.103:50038) 2023-07-02T09:04:38.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f979a59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f979c133360 0x7f979c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56482/0 (socket says 172.21.15.103:56482) 2023-07-02T09:04:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f979ad9d700 1 -- 172.21.15.103:0/4004367707 learned_addr learned my addr 172.21.15.103:0/4004367707 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f979a59c700 1 -- 172.21.15.103:0/4004367707 >> v1:172.21.15.103:6789/0 conn(0x7f979c132f50 legacy=0x7f979c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f979a59c700 1 -- 172.21.15.103:0/4004367707 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c132a20 con 0x7f979c133360 2023-07-02T09:04:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f979a59c700 1 --2- 172.21.15.103:0/4004367707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c133360 0x7f979c09cc90 crc :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a60b5eadd1f4e8 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f9799d9b700 1 -- 172.21.15.103:0/4004367707 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f978c009cd0 con 0x7f979c133360 2023-07-02T09:04:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f9799d9b700 1 -- 172.21.15.103:0/4004367707 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f978c009e30 con 0x7f979c133360 2023-07-02T09:04:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.667+0000 7f9799d9b700 1 -- 172.21.15.103:0/4004367707 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f978c00e360 con 0x7f979c133360 2023-07-02T09:04:38.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.668+0000 7f97a11c1700 1 -- 172.21.15.103:0/4004367707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c133360 msgr2=0x7f979c09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:38.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.668+0000 7f97a11c1700 1 --2- 172.21.15.103:0/4004367707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c133360 0x7f979c09cc90 crc :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:38.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.668+0000 7f97a11c1700 1 -- 172.21.15.103:0/4004367707 shutdown_connections 2023-07-02T09:04:38.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.668+0000 7f97a11c1700 1 --2- 172.21.15.103:0/4004367707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c133360 0x7f979c09cc90 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:38.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.668+0000 7f97a11c1700 1 -- 172.21.15.103:0/4004367707 >> 172.21.15.103:0/4004367707 conn(0x7f979c0a2080 msgr2=0x7f979c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:38.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.668+0000 7f97a11c1700 1 -- 172.21.15.103:0/4004367707 shutdown_connections 2023-07-02T09:04:38.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.668+0000 7f97a11c1700 1 -- 172.21.15.103:0/4004367707 wait complete. 2023-07-02T09:04:38.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.668+0000 7f97a11c1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:38.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.670+0000 7f97a11c1700 1 Processor -- start 2023-07-02T09:04:38.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.670+0000 7f97a11c1700 1 -- start start 2023-07-02T09:04:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.670+0000 7f97a11c1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c132f50 0x7f979c12bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.670+0000 7f97a11c1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c06d260 con 0x7f979c132f50 2023-07-02T09:04:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.670+0000 7f979ad9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c132f50 0x7f979c12bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.670+0000 7f979ad9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c132f50 0x7f979c12bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56490/0 (socket says 172.21.15.103:56490) 2023-07-02T09:04:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.670+0000 7f979ad9d700 1 -- 172.21.15.103:0/3108881645 learned_addr learned my addr 172.21.15.103:0/3108881645 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.671+0000 7f979ad9d700 1 -- 172.21.15.103:0/3108881645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c12bb90 con 0x7f979c132f50 2023-07-02T09:04:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.671+0000 7f979ad9d700 1 --2- 172.21.15.103:0/3108881645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c132f50 0x7f979c12bea0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.671+0000 7f97837fe700 1 -- 172.21.15.103:0/3108881645 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9790005a60 con 0x7f979c132f50 2023-07-02T09:04:38.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.671+0000 7f97837fe700 1 -- 172.21.15.103:0/3108881645 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f9790005bc0 con 0x7f979c132f50 2023-07-02T09:04:38.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.671+0000 7f97837fe700 1 -- 172.21.15.103:0/3108881645 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f9790005e60 con 0x7f979c132f50 2023-07-02T09:04:38.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.671+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c06f730 con 0x7f979c132f50 2023-07-02T09:04:38.674 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.671+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c06f490 con 0x7f979c132f50 2023-07-02T09:04:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.672+0000 7f97837fe700 1 -- 172.21.15.103:0/3108881645 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f97900070b0 con 0x7f979c132f50 2023-07-02T09:04:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.672+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979c061ea0 con 0x7f979c132f50 2023-07-02T09:04:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.672+0000 7f97837fe700 1 --2- 172.21.15.103:0/3108881645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97880377c0 0x7f9788039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.672+0000 7f97837fe700 1 -- 172.21.15.103:0/3108881645 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f97900469f0 con 0x7f979c132f50 2023-07-02T09:04:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.673+0000 7f979a59c700 1 --2- 172.21.15.103:0/3108881645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97880377c0 0x7f9788039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.673+0000 7f979a59c700 1 --2- 172.21.15.103:0/3108881645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97880377c0 0x7f9788039cd0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:38.677 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.677+0000 7f97837fe700 1 -- 172.21.15.103:0/3108881645 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f979000a4a0 con 0x7f979c132f50 2023-07-02T09:04:38.866 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.866+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f979c06f490 con 0x7f979c132f50 2023-07-02T09:04:38.867 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.866+0000 7f97837fe700 1 -- 172.21.15.103:0/3108881645 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f979c06f490 con 0x7f979c132f50 2023-07-02T09:04:38.867 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:38.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97880377c0 msgr2=0x7f9788039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:38.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 --2- 172.21.15.103:0/3108881645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97880377c0 0x7f9788039cd0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:38.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c132f50 msgr2=0x7f979c12bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:38.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 --2- 172.21.15.103:0/3108881645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c132f50 0x7f979c12bea0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:38.869 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 shutdown_connections 2023-07-02T09:04:38.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 --2- 172.21.15.103:0/3108881645 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f97880377c0 0x7f9788039cd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:38.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 --2- 172.21.15.103:0/3108881645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f979c132f50 0x7f979c12bea0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:38.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 >> 172.21.15.103:0/3108881645 conn(0x7f979c0a2080 msgr2=0x7f979c1bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:38.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 shutdown_connections 2023-07-02T09:04:38.870 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:38.869+0000 7f97a11c1700 1 -- 172.21.15.103:0/3108881645 wait complete. 2023-07-02T09:04:38.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:39.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:40.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.024+0000 7fed3b7c9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:40.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.029+0000 7fed3b7c9700 1 Processor -- start 2023-07-02T09:04:40.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.029+0000 7fed3b7c9700 1 -- start start 2023-07-02T09:04:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed3b7c9700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fed34132030 0x7fed34134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed3b7c9700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3406df60 con 0x7fed3412f250 2023-07-02T09:04:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed3b7c9700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3409a710 con 0x7fed34132030 2023-07-02T09:04:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed38d64700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fed34132030 0x7fed34134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed39565700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fed3412f250 0x7fed3412f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50044/0 (socket says 172.21.15.103:50044) 2023-07-02T09:04:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed39565700 1 -- 172.21.15.103:0/2171777053 learned_addr learned my addr 172.21.15.103:0/2171777053 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed38d64700 1 -- 172.21.15.103:0/2171777053 >> v1:172.21.15.103:6789/0 conn(0x7fed3412f250 legacy=0x7fed3412f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed38d64700 1 -- 172.21.15.103:0/2171777053 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3412ed20 con 0x7fed34132030 2023-07-02T09:04:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed38d64700 1 --2- 172.21.15.103:0/2171777053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed34132030 0x7fed34134450 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75a899b9499c6d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed2bfff700 1 -- 172.21.15.103:0/2171777053 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fed2400ac60 con 0x7fed34132030 2023-07-02T09:04:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed2bfff700 1 -- 172.21.15.103:0/2171777053 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fed2400adc0 con 0x7fed34132030 2023-07-02T09:04:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.030+0000 7fed2bfff700 1 -- 172.21.15.103:0/2171777053 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fed2400f470 con 0x7fed34132030 2023-07-02T09:04:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.031+0000 7fed3b7c9700 1 -- 172.21.15.103:0/2171777053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed34132030 msgr2=0x7fed34134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.031+0000 7fed3b7c9700 1 --2- 172.21.15.103:0/2171777053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed34132030 0x7fed34134450 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.031+0000 7fed3b7c9700 1 -- 172.21.15.103:0/2171777053 shutdown_connections 2023-07-02T09:04:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.031+0000 7fed3b7c9700 1 --2- 172.21.15.103:0/2171777053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed34132030 0x7fed34134450 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.031+0000 7fed3b7c9700 1 -- 172.21.15.103:0/2171777053 >> 172.21.15.103:0/2171777053 conn(0x7fed3409d810 msgr2=0x7fed3409dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.031+0000 7fed3b7c9700 1 -- 172.21.15.103:0/2171777053 shutdown_connections 2023-07-02T09:04:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.031+0000 7fed3b7c9700 1 -- 172.21.15.103:0/2171777053 wait complete. 2023-07-02T09:04:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.031+0000 7fed3b7c9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.032+0000 7fed3b7c9700 1 Processor -- start 2023-07-02T09:04:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.032+0000 7fed3b7c9700 1 -- start start 2023-07-02T09:04:40.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.033+0000 7fed3b7c9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed3412f250 0x7fed34117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:40.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.033+0000 7fed3b7c9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed34128ef0 con 0x7fed3412f250 2023-07-02T09:04:40.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.033+0000 7fed39565700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed3412f250 0x7fed34117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:40.036 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.033+0000 7fed39565700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed3412f250 0x7fed34117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56512/0 (socket says 172.21.15.103:56512) 2023-07-02T09:04:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.033+0000 7fed39565700 1 -- 172.21.15.103:0/621417875 learned_addr learned my addr 172.21.15.103:0/621417875 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.033+0000 7fed39565700 1 -- 172.21.15.103:0/621417875 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3411b6c0 con 0x7fed3412f250 2023-07-02T09:04:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.033+0000 7fed39565700 1 --2- 172.21.15.103:0/621417875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed3412f250 0x7fed34117360 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.034+0000 7fed29ffb700 1 -- 172.21.15.103:0/621417875 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fed30005a60 con 0x7fed3412f250 2023-07-02T09:04:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.034+0000 7fed29ffb700 1 -- 172.21.15.103:0/621417875 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fed30005bc0 con 0x7fed3412f250 2023-07-02T09:04:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.034+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed34118f30 con 0x7fed3412f250 2023-07-02T09:04:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.034+0000 7fed29ffb700 1 -- 172.21.15.103:0/621417875 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fed30005e60 con 0x7fed3412f250 2023-07-02T09:04:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.034+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed34119810 con 0x7fed3412f250 2023-07-02T09:04:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.034+0000 7fed29ffb700 1 -- 172.21.15.103:0/621417875 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fed300070b0 con 0x7fed3412f250 2023-07-02T09:04:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.035+0000 7fed29ffb700 1 --2- 172.21.15.103:0/621417875 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fed180377c0 0x7fed18039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.035+0000 7fed29ffb700 1 -- 172.21.15.103:0/621417875 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fed30046a90 con 0x7fed3412f250 2023-07-02T09:04:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.035+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed34128cf0 con 0x7fed3412f250 2023-07-02T09:04:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.035+0000 7fed38d64700 1 --2- 172.21.15.103:0/621417875 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fed180377c0 0x7fed18039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.035+0000 7fed38d64700 1 --2- 172.21.15.103:0/621417875 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fed180377c0 0x7fed18039cd0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:40.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.039+0000 7fed29ffb700 1 -- 172.21.15.103:0/621417875 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fed3000a640 con 0x7fed3412f250 2023-07-02T09:04:40.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.229+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fed34061ea0 con 0x7fed3412f250 2023-07-02T09:04:40.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.229+0000 7fed29ffb700 1 -- 172.21.15.103:0/621417875 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fed3000a640 con 0x7fed3412f250 2023-07-02T09:04:40.230 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:40.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fed180377c0 msgr2=0x7fed18039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:40.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 --2- 172.21.15.103:0/621417875 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fed180377c0 0x7fed18039cd0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:40.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed3412f250 msgr2=0x7fed34117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 --2- 172.21.15.103:0/621417875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed3412f250 0x7fed34117360 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 shutdown_connections 2023-07-02T09:04:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 --2- 172.21.15.103:0/621417875 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fed180377c0 0x7fed18039cd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 --2- 172.21.15.103:0/621417875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed3412f250 0x7fed34117360 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 >> 172.21.15.103:0/621417875 conn(0x7fed3409d810 msgr2=0x7fed3412e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 shutdown_connections 2023-07-02T09:04:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:40.232+0000 7fed3b7c9700 1 -- 172.21.15.103:0/621417875 wait complete. 2023-07-02T09:04:40.245 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:41.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:41.391 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.387+0000 7fd6d3b74700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:41.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.392+0000 7fd6d3b74700 1 Processor -- start 2023-07-02T09:04:41.392 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.392+0000 7fd6d3b74700 1 -- start start 2023-07-02T09:04:41.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d3b74700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd6cc133360 0x7fd6cc09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:41.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d3b74700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc09e260 con 0x7fd6cc132f50 2023-07-02T09:04:41.393 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d3b74700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc0a1640 con 0x7fd6cc133360 2023-07-02T09:04:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d110f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd6cc133360 0x7fd6cc09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d1910700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd6cc132f50 0x7fd6cc09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50060/0 (socket says 172.21.15.103:50060) 2023-07-02T09:04:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d110f700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd6cc133360 0x7fd6cc09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56518/0 (socket says 172.21.15.103:56518) 2023-07-02T09:04:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d1910700 1 -- 172.21.15.103:0/3084378290 learned_addr learned my addr 172.21.15.103:0/3084378290 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d110f700 1 -- 172.21.15.103:0/3084378290 >> v1:172.21.15.103:6789/0 conn(0x7fd6cc132f50 legacy=0x7fd6cc09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:41.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d110f700 1 -- 172.21.15.103:0/3084378290 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc132a20 con 0x7fd6cc133360 2023-07-02T09:04:41.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d090e700 1 -- 172.21.15.103:0/3084378290 <== mon.0 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142472374 0 0) 0x7fd6cc09e260 con 0x7fd6cc132f50 2023-07-02T09:04:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.393+0000 7fd6d110f700 1 --2- 172.21.15.103:0/3084378290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc133360 0x7fd6cc09cc90 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232be3dda04adde7 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d090e700 1 -- 172.21.15.103:0/3084378290 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd6c8009cd0 con 0x7fd6cc133360 2023-07-02T09:04:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d090e700 1 -- 172.21.15.103:0/3084378290 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd6c8009e30 con 0x7fd6cc133360 2023-07-02T09:04:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d090e700 1 -- 172.21.15.103:0/3084378290 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd6c800e360 con 0x7fd6cc133360 2023-07-02T09:04:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d3b74700 1 -- 172.21.15.103:0/3084378290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc133360 msgr2=0x7fd6cc09cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d3b74700 1 --2- 172.21.15.103:0/3084378290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc133360 0x7fd6cc09cc90 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d3b74700 1 -- 172.21.15.103:0/3084378290 shutdown_connections 2023-07-02T09:04:41.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d3b74700 1 --2- 172.21.15.103:0/3084378290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc133360 0x7fd6cc09cc90 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:41.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d3b74700 1 -- 172.21.15.103:0/3084378290 >> 172.21.15.103:0/3084378290 conn(0x7fd6cc0a2080 msgr2=0x7fd6cc0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:41.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d3b74700 1 -- 172.21.15.103:0/3084378290 shutdown_connections 2023-07-02T09:04:41.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.394+0000 7fd6d3b74700 1 -- 172.21.15.103:0/3084378290 wait complete. 2023-07-02T09:04:41.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.395+0000 7fd6d3b74700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.396+0000 7fd6d3b74700 1 Processor -- start 2023-07-02T09:04:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.396+0000 7fd6d3b74700 1 -- start start 2023-07-02T09:04:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d3b74700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc132f50 0x7fd6cc1ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d3b74700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc06d370 con 0x7fd6cc132f50 2023-07-02T09:04:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d1910700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc132f50 0x7fd6cc1ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d1910700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc132f50 0x7fd6cc1ba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56520/0 (socket says 172.21.15.103:56520) 2023-07-02T09:04:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d1910700 1 -- 172.21.15.103:0/1163878272 learned_addr learned my addr 172.21.15.103:0/1163878272 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d1910700 1 -- 172.21.15.103:0/1163878272 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc06dde0 con 0x7fd6cc132f50 2023-07-02T09:04:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d1910700 1 --2- 172.21.15.103:0/1163878272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc132f50 0x7fd6cc1ba9b0 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6c27fc700 1 -- 172.21.15.103:0/1163878272 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd6bc004b80 con 0x7fd6cc132f50 2023-07-02T09:04:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6c27fc700 1 -- 172.21.15.103:0/1163878272 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd6bc004ce0 con 0x7fd6cc132f50 2023-07-02T09:04:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc1ba4f0 con 0x7fd6cc132f50 2023-07-02T09:04:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6c27fc700 1 -- 172.21.15.103:0/1163878272 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd6bc00c4f0 con 0x7fd6cc132f50 2023-07-02T09:04:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.397+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc071e40 con 0x7fd6cc132f50 2023-07-02T09:04:41.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.398+0000 7fd6c27fc700 1 -- 172.21.15.103:0/1163878272 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fd6bc00c650 con 0x7fd6cc132f50 2023-07-02T09:04:41.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.398+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc061ea0 con 0x7fd6cc132f50 2023-07-02T09:04:41.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.399+0000 7fd6c27fc700 1 --2- 172.21.15.103:0/1163878272 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6b00377c0 0x7fd6b0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:41.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.399+0000 7fd6c27fc700 1 -- 172.21.15.103:0/1163878272 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fd6bc046970 con 0x7fd6cc132f50 2023-07-02T09:04:41.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.399+0000 7fd6d110f700 1 --2- 172.21.15.103:0/1163878272 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6b00377c0 0x7fd6b0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:41.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.399+0000 7fd6d110f700 1 --2- 172.21.15.103:0/1163878272 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6b00377c0 0x7fd6b0039cd0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:41.403 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.403+0000 7fd6c27fc700 1 -- 172.21.15.103:0/1163878272 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fd6bc009c80 con 0x7fd6cc132f50 2023-07-02T09:04:41.595 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.595+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd6cc0725b0 con 0x7fd6cc132f50 2023-07-02T09:04:41.596 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.595+0000 7fd6c27fc700 1 -- 172.21.15.103:0/1163878272 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fd6bc0098a0 con 0x7fd6cc132f50 2023-07-02T09:04:41.596 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:41.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.597+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6b00377c0 msgr2=0x7fd6b0039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:41.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.597+0000 7fd6d3b74700 1 --2- 172.21.15.103:0/1163878272 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6b00377c0 0x7fd6b0039cd0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:41.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.597+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc132f50 msgr2=0x7fd6cc1ba9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:41.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.597+0000 7fd6d3b74700 1 --2- 172.21.15.103:0/1163878272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc132f50 0x7fd6cc1ba9b0 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:41.598 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.597+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 shutdown_connections 2023-07-02T09:04:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.597+0000 7fd6d3b74700 1 --2- 172.21.15.103:0/1163878272 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fd6b00377c0 0x7fd6b0039cd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.597+0000 7fd6d3b74700 1 --2- 172.21.15.103:0/1163878272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6cc132f50 0x7fd6cc1ba9b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.598+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 >> 172.21.15.103:0/1163878272 conn(0x7fd6cc0a2080 msgr2=0x7fd6cc0a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.598+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 shutdown_connections 2023-07-02T09:04:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:41.598+0000 7fd6d3b74700 1 -- 172.21.15.103:0/1163878272 wait complete. 2023-07-02T09:04:41.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:42.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:42.748 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.743+0000 7f1ea9ff1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:42.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.749+0000 7f1ea9ff1700 1 Processor -- start 2023-07-02T09:04:42.749 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.749+0000 7f1ea9ff1700 1 -- start start 2023-07-02T09:04:42.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.749+0000 7f1ea9ff1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ea4133360 0x7f1ea409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:42.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.749+0000 7f1ea9ff1700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea409e260 con 0x7f1ea4132f50 2023-07-02T09:04:42.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.749+0000 7f1ea9ff1700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea40a1640 con 0x7f1ea4133360 2023-07-02T09:04:42.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea37fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1ea4132f50 0x7f1ea409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:50076/0 (socket says 172.21.15.103:50076) 2023-07-02T09:04:42.750 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea37fe700 1 -- 172.21.15.103:0/1905140751 learned_addr learned my addr 172.21.15.103:0/1905140751 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:42.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea2ffd700 1 --2- 172.21.15.103:0/1905140751 >> v2:172.21.15.103:3300/0 conn(0x7f1ea4133360 0x7f1ea409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:42.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea2ffd700 1 -- 172.21.15.103:0/1905140751 >> v1:172.21.15.103:6789/0 conn(0x7f1ea4132f50 legacy=0x7f1ea409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:42.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea2ffd700 1 -- 172.21.15.103:0/1905140751 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4132a20 con 0x7f1ea4133360 2023-07-02T09:04:42.751 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea2ffd700 1 --2- 172.21.15.103:0/1905140751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4133360 0x7f1ea409cc90 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f80be603ca32dd5 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:42.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea27fc700 1 -- 172.21.15.103:0/1905140751 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1e98009cd0 con 0x7f1ea4133360 2023-07-02T09:04:42.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea27fc700 1 -- 172.21.15.103:0/1905140751 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1e98009e30 con 0x7f1ea4133360 2023-07-02T09:04:42.752 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.750+0000 7f1ea27fc700 1 -- 172.21.15.103:0/1905140751 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1e9800e360 con 0x7f1ea4133360 2023-07-02T09:04:42.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.751+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1905140751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4133360 msgr2=0x7f1ea409cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:42.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.751+0000 7f1ea9ff1700 1 --2- 172.21.15.103:0/1905140751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4133360 0x7f1ea409cc90 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:42.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.751+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1905140751 shutdown_connections 2023-07-02T09:04:42.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.751+0000 7f1ea9ff1700 1 --2- 172.21.15.103:0/1905140751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4133360 0x7f1ea409cc90 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:42.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.751+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1905140751 >> 172.21.15.103:0/1905140751 conn(0x7f1ea40a2080 msgr2=0x7f1ea40a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:42.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.751+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1905140751 shutdown_connections 2023-07-02T09:04:42.753 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.751+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1905140751 wait complete. 2023-07-02T09:04:42.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.751+0000 7f1ea9ff1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:42.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.752+0000 7f1ea9ff1700 1 Processor -- start 2023-07-02T09:04:42.754 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.752+0000 7f1ea9ff1700 1 -- start start 2023-07-02T09:04:42.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.753+0000 7f1ea9ff1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4132f50 0x7f1ea412bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:42.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.753+0000 7f1ea9ff1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea406d260 con 0x7f1ea4132f50 2023-07-02T09:04:42.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.753+0000 7f1ea37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4132f50 0x7f1ea412bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:42.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.753+0000 7f1ea37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4132f50 0x7f1ea412bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56546/0 (socket says 172.21.15.103:56546) 2023-07-02T09:04:42.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.753+0000 7f1ea37fe700 1 -- 172.21.15.103:0/1448658173 learned_addr learned my addr 172.21.15.103:0/1448658173 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:42.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.753+0000 7f1ea37fe700 1 -- 172.21.15.103:0/1448658173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea412bb90 con 0x7f1ea4132f50 2023-07-02T09:04:42.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.753+0000 7f1ea37fe700 1 --2- 172.21.15.103:0/1448658173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4132f50 0x7f1ea412bea0 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:42.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.754+0000 7f1e8bfff700 1 -- 172.21.15.103:0/1448658173 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1e94005a60 con 0x7f1ea4132f50 2023-07-02T09:04:42.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.754+0000 7f1e8bfff700 1 -- 172.21.15.103:0/1448658173 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1e94005bc0 con 0x7f1ea4132f50 2023-07-02T09:04:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.754+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea406f730 con 0x7f1ea4132f50 2023-07-02T09:04:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.754+0000 7f1e8bfff700 1 -- 172.21.15.103:0/1448658173 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1e94005e60 con 0x7f1ea4132f50 2023-07-02T09:04:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.754+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea406f490 con 0x7f1ea4132f50 2023-07-02T09:04:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.754+0000 7f1e8bfff700 1 -- 172.21.15.103:0/1448658173 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f1e940070b0 con 0x7f1ea4132f50 2023-07-02T09:04:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.755+0000 7f1e8bfff700 1 --2- 172.21.15.103:0/1448658173 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e900377c0 0x7f1e90039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.755+0000 7f1e8bfff700 1 -- 172.21.15.103:0/1448658173 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f1e940469f0 con 0x7f1ea4132f50 2023-07-02T09:04:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.755+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea406f490 con 0x7f1ea4132f50 2023-07-02T09:04:42.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.755+0000 7f1ea2ffd700 1 --2- 172.21.15.103:0/1448658173 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e900377c0 0x7f1e90039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:42.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.755+0000 7f1ea2ffd700 1 --2- 172.21.15.103:0/1448658173 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e900377c0 0x7f1e90039cd0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:42.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.760+0000 7f1e8bfff700 1 -- 172.21.15.103:0/1448658173 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f1ea406f490 con 0x7f1ea4132f50 2023-07-02T09:04:42.952 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.952+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1ea406f490 con 0x7f1ea4132f50 2023-07-02T09:04:42.952 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.952+0000 7f1e8bfff700 1 -- 172.21.15.103:0/1448658173 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f1ea406f490 con 0x7f1ea4132f50 2023-07-02T09:04:42.953 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:42.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.954+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e900377c0 msgr2=0x7f1e90039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:42.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.954+0000 7f1ea9ff1700 1 --2- 172.21.15.103:0/1448658173 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e900377c0 0x7f1e90039cd0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:42.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.954+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4132f50 msgr2=0x7f1ea412bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:42.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.954+0000 7f1ea9ff1700 1 --2- 172.21.15.103:0/1448658173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4132f50 0x7f1ea412bea0 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:42.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.954+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 shutdown_connections 2023-07-02T09:04:42.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.954+0000 7f1ea9ff1700 1 --2- 172.21.15.103:0/1448658173 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f1e900377c0 0x7f1e90039cd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:42.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.954+0000 7f1ea9ff1700 1 --2- 172.21.15.103:0/1448658173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea4132f50 0x7f1ea412bea0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:42.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.954+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 >> 172.21.15.103:0/1448658173 conn(0x7f1ea40a2080 msgr2=0x7f1ea41bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:42.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.955+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 shutdown_connections 2023-07-02T09:04:42.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:42.955+0000 7f1ea9ff1700 1 -- 172.21.15.103:0/1448658173 wait complete. 2023-07-02T09:04:42.968 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:43.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~297s 2023-07-02T09:04:43.969 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:44.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.098+0000 7fb9e4c50700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:44.103 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.103+0000 7fb9e4c50700 1 Processor -- start 2023-07-02T09:04:44.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.103+0000 7fb9e4c50700 1 -- start start 2023-07-02T09:04:44.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.104+0000 7fb9e4c50700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb9e0132f50 0x7fb9e0133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:44.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.104+0000 7fb9e4c50700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e009e260 con 0x7fb9e009c580 2023-07-02T09:04:44.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.104+0000 7fb9e4c50700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e00a1640 con 0x7fb9e0132f50 2023-07-02T09:04:44.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.104+0000 7fb9de59c700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb9e0132f50 0x7fb9e0133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:44.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.104+0000 7fb9ddd9b700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb9e009c580 0x7fb9e009c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:45066/0 (socket says 172.21.15.103:45066) 2023-07-02T09:04:44.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.104+0000 7fb9ddd9b700 1 -- 172.21.15.103:0/2769370692 learned_addr learned my addr 172.21.15.103:0/2769370692 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:44.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.105+0000 7fb9de59c700 1 -- 172.21.15.103:0/2769370692 >> v1:172.21.15.103:6789/0 conn(0x7fb9e009c580 legacy=0x7fb9e009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:44.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.105+0000 7fb9de59c700 1 -- 172.21.15.103:0/2769370692 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e0132a20 con 0x7fb9e0132f50 2023-07-02T09:04:44.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.105+0000 7fb9de59c700 1 --2- 172.21.15.103:0/2769370692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e0132f50 0x7fb9e0133360 crc :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c0c7dc90018d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:44.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.105+0000 7fb9dd59a700 1 -- 172.21.15.103:0/2769370692 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb9d4009cd0 con 0x7fb9e0132f50 2023-07-02T09:04:44.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.105+0000 7fb9dd59a700 1 -- 172.21.15.103:0/2769370692 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb9d4009e30 con 0x7fb9e0132f50 2023-07-02T09:04:44.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.105+0000 7fb9dd59a700 1 -- 172.21.15.103:0/2769370692 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb9d400e360 con 0x7fb9e0132f50 2023-07-02T09:04:44.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.106+0000 7fb9e4c50700 1 -- 172.21.15.103:0/2769370692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e0132f50 msgr2=0x7fb9e0133360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:44.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.106+0000 7fb9e4c50700 1 --2- 172.21.15.103:0/2769370692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e0132f50 0x7fb9e0133360 crc :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:44.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.106+0000 7fb9e4c50700 1 -- 172.21.15.103:0/2769370692 shutdown_connections 2023-07-02T09:04:44.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.106+0000 7fb9e4c50700 1 --2- 172.21.15.103:0/2769370692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e0132f50 0x7fb9e0133360 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:44.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.106+0000 7fb9e4c50700 1 -- 172.21.15.103:0/2769370692 >> 172.21.15.103:0/2769370692 conn(0x7fb9e00a2080 msgr2=0x7fb9e00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:44.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.106+0000 7fb9e4c50700 1 -- 172.21.15.103:0/2769370692 shutdown_connections 2023-07-02T09:04:44.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.106+0000 7fb9e4c50700 1 -- 172.21.15.103:0/2769370692 wait complete. 2023-07-02T09:04:44.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.106+0000 7fb9e4c50700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:44.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.107+0000 7fb9e4c50700 1 Processor -- start 2023-07-02T09:04:44.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.107+0000 7fb9e4c50700 1 -- start start 2023-07-02T09:04:44.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9e4c50700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e009c580 0x7fb9e012b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:44.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9e4c50700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e006f9d0 con 0x7fb9e009c580 2023-07-02T09:04:44.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9de59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e009c580 0x7fb9e012b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:44.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9de59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e009c580 0x7fb9e012b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41322/0 (socket says 172.21.15.103:41322) 2023-07-02T09:04:44.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9de59c700 1 -- 172.21.15.103:0/3638875792 learned_addr learned my addr 172.21.15.103:0/3638875792 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:44.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9de59c700 1 -- 172.21.15.103:0/3638875792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e0133a00 con 0x7fb9e009c580 2023-07-02T09:04:44.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9de59c700 1 --2- 172.21.15.103:0/3638875792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e009c580 0x7fb9e012b7b0 crc :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:44.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9ceffd700 1 -- 172.21.15.103:0/3638875792 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb9d4013530 con 0x7fb9e009c580 2023-07-02T09:04:44.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9ceffd700 1 -- 172.21.15.103:0/3638875792 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb9d4013690 con 0x7fb9e009c580 2023-07-02T09:04:44.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e006d160 con 0x7fb9e009c580 2023-07-02T09:04:44.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9ceffd700 1 -- 172.21.15.103:0/3638875792 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb9d4013930 con 0x7fb9e009c580 2023-07-02T09:04:44.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.108+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e006da70 con 0x7fb9e009c580 2023-07-02T09:04:44.111 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.109+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e0061ea0 con 0x7fb9e009c580 2023-07-02T09:04:44.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.109+0000 7fb9ceffd700 1 -- 172.21.15.103:0/3638875792 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fb9d4013b30 con 0x7fb9e009c580 2023-07-02T09:04:44.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.109+0000 7fb9ceffd700 1 --2- 172.21.15.103:0/3638875792 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb9c0037770 0x7fb9c0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:44.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.109+0000 7fb9ceffd700 1 -- 172.21.15.103:0/3638875792 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fb9d4018040 con 0x7fb9e009c580 2023-07-02T09:04:44.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.110+0000 7fb9ddd9b700 1 --2- 172.21.15.103:0/3638875792 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb9c0037770 0x7fb9c0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:44.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.110+0000 7fb9ddd9b700 1 --2- 172.21.15.103:0/3638875792 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb9c0037770 0x7fb9c0039c80 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:44.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.114+0000 7fb9ceffd700 1 -- 172.21.15.103:0/3638875792 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fb9d4018e00 con 0x7fb9e009c580 2023-07-02T09:04:44.306 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.306+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb9e00725b0 con 0x7fb9e009c580 2023-07-02T09:04:44.307 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.306+0000 7fb9ceffd700 1 -- 172.21.15.103:0/3638875792 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fb9d4010e60 con 0x7fb9e009c580 2023-07-02T09:04:44.307 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:44.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.308+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb9c0037770 msgr2=0x7fb9c0039c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:44.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.308+0000 7fb9e4c50700 1 --2- 172.21.15.103:0/3638875792 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb9c0037770 0x7fb9c0039c80 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:44.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.309+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e009c580 msgr2=0x7fb9e012b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:44.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.309+0000 7fb9e4c50700 1 --2- 172.21.15.103:0/3638875792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e009c580 0x7fb9e012b7b0 crc :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:44.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.309+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 shutdown_connections 2023-07-02T09:04:44.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.309+0000 7fb9e4c50700 1 --2- 172.21.15.103:0/3638875792 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fb9c0037770 0x7fb9c0039c80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:44.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.309+0000 7fb9e4c50700 1 --2- 172.21.15.103:0/3638875792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9e009c580 0x7fb9e012b7b0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:44.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.309+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 >> 172.21.15.103:0/3638875792 conn(0x7fb9e00a2080 msgr2=0x7fb9e01babc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:44.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.309+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 shutdown_connections 2023-07-02T09:04:44.310 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:44.309+0000 7fb9e4c50700 1 -- 172.21.15.103:0/3638875792 wait complete. 2023-07-02T09:04:44.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:45.322 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:45.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.462+0000 7f35bcf37700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:45.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.467+0000 7f35bcf37700 1 Processor -- start 2023-07-02T09:04:45.467 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.467+0000 7f35bcf37700 1 -- start start 2023-07-02T09:04:45.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35bcf37700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35b8132030 0x7f35b8134450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:45.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35bcf37700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b806df60 con 0x7f35b812f250 2023-07-02T09:04:45.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35bcf37700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b809a710 con 0x7f35b8132030 2023-07-02T09:04:45.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35b5d9b700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35b8132030 0x7f35b8134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:45.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35b659c700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f35b812f250 0x7f35b812f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:45082/0 (socket says 172.21.15.103:45082) 2023-07-02T09:04:45.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35b659c700 1 -- 172.21.15.103:0/692834041 learned_addr learned my addr 172.21.15.103:0/692834041 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:45.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35b5d9b700 1 -- 172.21.15.103:0/692834041 >> v1:172.21.15.103:6789/0 conn(0x7f35b812f250 legacy=0x7f35b812f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:45.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35b5d9b700 1 -- 172.21.15.103:0/692834041 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b812ed20 con 0x7f35b8132030 2023-07-02T09:04:45.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.468+0000 7f35b5d9b700 1 --2- 172.21.15.103:0/692834041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b8132030 0x7f35b8134450 crc :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f818eb6f77b1ed5a server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:45.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35b559a700 1 -- 172.21.15.103:0/692834041 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f35a800ac60 con 0x7f35b8132030 2023-07-02T09:04:45.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35b559a700 1 -- 172.21.15.103:0/692834041 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f35a800adc0 con 0x7f35b8132030 2023-07-02T09:04:45.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35b559a700 1 -- 172.21.15.103:0/692834041 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f35a800f470 con 0x7f35b8132030 2023-07-02T09:04:45.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35bcf37700 1 -- 172.21.15.103:0/692834041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b8132030 msgr2=0x7f35b8134450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:45.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35bcf37700 1 --2- 172.21.15.103:0/692834041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b8132030 0x7f35b8134450 crc :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:45.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35bcf37700 1 -- 172.21.15.103:0/692834041 shutdown_connections 2023-07-02T09:04:45.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35bcf37700 1 --2- 172.21.15.103:0/692834041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b8132030 0x7f35b8134450 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:45.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35bcf37700 1 -- 172.21.15.103:0/692834041 >> 172.21.15.103:0/692834041 conn(0x7f35b809d810 msgr2=0x7f35b809dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:45.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35bcf37700 1 -- 172.21.15.103:0/692834041 shutdown_connections 2023-07-02T09:04:45.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.469+0000 7f35bcf37700 1 -- 172.21.15.103:0/692834041 wait complete. 2023-07-02T09:04:45.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.470+0000 7f35bcf37700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:45.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.471+0000 7f35bcf37700 1 Processor -- start 2023-07-02T09:04:45.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.471+0000 7f35bcf37700 1 -- start start 2023-07-02T09:04:45.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.471+0000 7f35bcf37700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b812f250 0x7f35b81c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:45.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.471+0000 7f35bcf37700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b811b9d0 con 0x7f35b812f250 2023-07-02T09:04:45.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35b659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b812f250 0x7f35b81c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:45.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35b659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b812f250 0x7f35b81c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41346/0 (socket says 172.21.15.103:41346) 2023-07-02T09:04:45.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35b659c700 1 -- 172.21.15.103:0/4245669588 learned_addr learned my addr 172.21.15.103:0/4245669588 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:45.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35b659c700 1 -- 172.21.15.103:0/4245669588 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b8071340 con 0x7f35b812f250 2023-07-02T09:04:45.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35b659c700 1 --2- 172.21.15.103:0/4245669588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b812f250 0x7f35b81c7250 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:45.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35a6ffd700 1 -- 172.21.15.103:0/4245669588 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f35ac005a60 con 0x7f35b812f250 2023-07-02T09:04:45.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35a6ffd700 1 -- 172.21.15.103:0/4245669588 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f35ac005bc0 con 0x7f35b812f250 2023-07-02T09:04:45.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b8071d60 con 0x7f35b812f250 2023-07-02T09:04:45.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35a6ffd700 1 -- 172.21.15.103:0/4245669588 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f35ac005e60 con 0x7f35b812f250 2023-07-02T09:04:45.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.472+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b8129060 con 0x7f35b812f250 2023-07-02T09:04:45.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.473+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b811bb10 con 0x7f35b812f250 2023-07-02T09:04:45.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.473+0000 7f35a6ffd700 1 -- 172.21.15.103:0/4245669588 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f35ac0070b0 con 0x7f35b812f250 2023-07-02T09:04:45.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.473+0000 7f35a6ffd700 1 --2- 172.21.15.103:0/4245669588 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f35980377c0 0x7f3598039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:45.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.473+0000 7f35a6ffd700 1 -- 172.21.15.103:0/4245669588 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f35ac046b30 con 0x7f35b812f250 2023-07-02T09:04:45.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.474+0000 7f35b5d9b700 1 --2- 172.21.15.103:0/4245669588 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f35980377c0 0x7f3598039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:45.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.474+0000 7f35b5d9b700 1 --2- 172.21.15.103:0/4245669588 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f35980377c0 0x7f3598039cd0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:45.478 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.478+0000 7f35a6ffd700 1 -- 172.21.15.103:0/4245669588 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f35ac00fdf0 con 0x7f35b812f250 2023-07-02T09:04:45.668 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.668+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f35b8061ea0 con 0x7f35b812f250 2023-07-02T09:04:45.669 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.668+0000 7f35a6ffd700 1 -- 172.21.15.103:0/4245669588 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f35ac00ddd0 con 0x7f35b812f250 2023-07-02T09:04:45.669 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:45.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f35980377c0 msgr2=0x7f3598039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:45.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 --2- 172.21.15.103:0/4245669588 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f35980377c0 0x7f3598039cd0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:45.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b812f250 msgr2=0x7f35b81c7250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:45.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 --2- 172.21.15.103:0/4245669588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b812f250 0x7f35b81c7250 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:45.671 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 shutdown_connections 2023-07-02T09:04:45.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 --2- 172.21.15.103:0/4245669588 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f35980377c0 0x7f3598039cd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:45.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 --2- 172.21.15.103:0/4245669588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35b812f250 0x7f35b81c7250 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:45.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 >> 172.21.15.103:0/4245669588 conn(0x7f35b809d810 msgr2=0x7f35b812e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:45.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.670+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 shutdown_connections 2023-07-02T09:04:45.672 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:45.671+0000 7f35bcf37700 1 -- 172.21.15.103:0/4245669588 wait complete. 2023-07-02T09:04:45.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:46.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:46.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.814+0000 7f61e5ee1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:46.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.820+0000 7f61e5ee1700 1 Processor -- start 2023-07-02T09:04:46.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.820+0000 7f61e5ee1700 1 -- start start 2023-07-02T09:04:46.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.820+0000 7f61e5ee1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f61e0133360 0x7f61e009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:46.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.820+0000 7f61e5ee1700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e009e260 con 0x7f61e0132f50 2023-07-02T09:04:46.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.820+0000 7f61e5ee1700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e00a1640 con 0x7f61e0133360 2023-07-02T09:04:46.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.820+0000 7f61deffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f61e0133360 0x7f61e009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.820+0000 7f61df7fe700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f61e0132f50 0x7f61e009c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:45088/0 (socket says 172.21.15.103:45088) 2023-07-02T09:04:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.820+0000 7f61df7fe700 1 -- 172.21.15.103:0/3252560174 learned_addr learned my addr 172.21.15.103:0/3252560174 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.821+0000 7f61deffd700 1 -- 172.21.15.103:0/3252560174 >> v1:172.21.15.103:6789/0 conn(0x7f61e0132f50 legacy=0x7f61e009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.821+0000 7f61deffd700 1 -- 172.21.15.103:0/3252560174 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e0132a20 con 0x7f61e0133360 2023-07-02T09:04:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.821+0000 7f61deffd700 1 --2- 172.21.15.103:0/3252560174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0133360 0x7f61e009cc90 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e10008660e66176 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.821+0000 7f61de7fc700 1 -- 172.21.15.103:0/3252560174 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f61c8009cd0 con 0x7f61e0133360 2023-07-02T09:04:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.821+0000 7f61de7fc700 1 -- 172.21.15.103:0/3252560174 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f61c8009e30 con 0x7f61e0133360 2023-07-02T09:04:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.821+0000 7f61de7fc700 1 -- 172.21.15.103:0/3252560174 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f61c800e360 con 0x7f61e0133360 2023-07-02T09:04:46.823 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.822+0000 7f61e5ee1700 1 -- 172.21.15.103:0/3252560174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0133360 msgr2=0x7f61e009cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:46.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.822+0000 7f61e5ee1700 1 --2- 172.21.15.103:0/3252560174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0133360 0x7f61e009cc90 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:46.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.822+0000 7f61e5ee1700 1 -- 172.21.15.103:0/3252560174 shutdown_connections 2023-07-02T09:04:46.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.822+0000 7f61e5ee1700 1 --2- 172.21.15.103:0/3252560174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0133360 0x7f61e009cc90 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:46.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.822+0000 7f61e5ee1700 1 -- 172.21.15.103:0/3252560174 >> 172.21.15.103:0/3252560174 conn(0x7f61e00a2080 msgr2=0x7f61e00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:46.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.822+0000 7f61e5ee1700 1 -- 172.21.15.103:0/3252560174 shutdown_connections 2023-07-02T09:04:46.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.822+0000 7f61e5ee1700 1 -- 172.21.15.103:0/3252560174 wait complete. 2023-07-02T09:04:46.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.822+0000 7f61e5ee1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:46.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.823+0000 7f61e5ee1700 1 Processor -- start 2023-07-02T09:04:46.825 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.823+0000 7f61e5ee1700 1 -- start start 2023-07-02T09:04:46.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61e5ee1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0132f50 0x7f61e012bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:46.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61e5ee1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e006d260 con 0x7f61e0132f50 2023-07-02T09:04:46.826 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61df7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0132f50 0x7f61e012bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:46.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61df7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0132f50 0x7f61e012bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41362/0 (socket says 172.21.15.103:41362) 2023-07-02T09:04:46.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61df7fe700 1 -- 172.21.15.103:0/2028869714 learned_addr learned my addr 172.21.15.103:0/2028869714 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:46.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61df7fe700 1 -- 172.21.15.103:0/2028869714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e012bb90 con 0x7f61e0132f50 2023-07-02T09:04:46.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61df7fe700 1 --2- 172.21.15.103:0/2028869714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0132f50 0x7f61e012bea0 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:46.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61d7fff700 1 -- 172.21.15.103:0/2028869714 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f61d0005a60 con 0x7f61e0132f50 2023-07-02T09:04:46.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61d7fff700 1 -- 172.21.15.103:0/2028869714 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f61d0005bc0 con 0x7f61e0132f50 2023-07-02T09:04:46.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e006f730 con 0x7f61e0132f50 2023-07-02T09:04:46.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61d7fff700 1 -- 172.21.15.103:0/2028869714 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f61d0005e60 con 0x7f61e0132f50 2023-07-02T09:04:46.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.824+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e006f490 con 0x7f61e0132f50 2023-07-02T09:04:46.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.825+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0061ea0 con 0x7f61e0132f50 2023-07-02T09:04:46.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.826+0000 7f61d7fff700 1 -- 172.21.15.103:0/2028869714 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7f61d00070b0 con 0x7f61e0132f50 2023-07-02T09:04:46.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.826+0000 7f61d7fff700 1 --2- 172.21.15.103:0/2028869714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f61c0037810 0x7f61c0039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:46.828 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.826+0000 7f61d7fff700 1 -- 172.21.15.103:0/2028869714 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7f61d0046cf0 con 0x7f61e0132f50 2023-07-02T09:04:46.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.826+0000 7f61deffd700 1 --2- 172.21.15.103:0/2028869714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f61c0037810 0x7f61c0039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:46.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.827+0000 7f61deffd700 1 --2- 172.21.15.103:0/2028869714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f61c0037810 0x7f61c0039d20 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:46.830 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:46.830+0000 7f61d7fff700 1 -- 172.21.15.103:0/2028869714 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7f61d000d4d0 con 0x7f61e0132f50 2023-07-02T09:04:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.019+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f61e006f490 con 0x7f61e0132f50 2023-07-02T09:04:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.019+0000 7f61d7fff700 1 -- 172.21.15.103:0/2028869714 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7f61e006f490 con 0x7f61e0132f50 2023-07-02T09:04:47.020 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:47.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f61c0037810 msgr2=0x7f61c0039d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:47.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 --2- 172.21.15.103:0/2028869714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f61c0037810 0x7f61c0039d20 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:47.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0132f50 msgr2=0x7f61e012bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:47.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 --2- 172.21.15.103:0/2028869714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0132f50 0x7f61e012bea0 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:47.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 shutdown_connections 2023-07-02T09:04:47.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 --2- 172.21.15.103:0/2028869714 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7f61c0037810 0x7f61c0039d20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:47.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 --2- 172.21.15.103:0/2028869714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61e0132f50 0x7f61e012bea0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:47.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 >> 172.21.15.103:0/2028869714 conn(0x7f61e00a2080 msgr2=0x7f61e01bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:47.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 shutdown_connections 2023-07-02T09:04:47.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:47.022+0000 7f61e5ee1700 1 -- 172.21.15.103:0/2028869714 wait complete. 2023-07-02T09:04:47.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:48.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:48.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.171+0000 7fadf20c1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:48.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.176+0000 7fadf20c1700 1 Processor -- start 2023-07-02T09:04:48.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.176+0000 7fadf20c1700 1 -- start start 2023-07-02T09:04:48.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadf20c1700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fadec12f250 0x7fadec12f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:48.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadf20c1700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec06df60 con 0x7fadec098a00 2023-07-02T09:04:48.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadf20c1700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadec09a710 con 0x7fadec12f250 2023-07-02T09:04:48.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadeb7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fadec12f250 0x7fadec12f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:48.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadeaffd700 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fadec098a00 0x7fadec098e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:45100/0 (socket says 172.21.15.103:45100) 2023-07-02T09:04:48.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadeaffd700 1 -- 172.21.15.103:0/2234204883 learned_addr learned my addr 172.21.15.103:0/2234204883 (peer_addr_for_me v1:172.21.15.103:0/0) 2023-07-02T09:04:48.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadeb7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fadec12f250 0x7fadec12f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41366/0 (socket says 172.21.15.103:41366) 2023-07-02T09:04:48.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadeb7fe700 1 -- 172.21.15.103:0/2234204883 >> v1:172.21.15.103:6789/0 conn(0x7fadec098a00 legacy=0x7fadec098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:48.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadeb7fe700 1 -- 172.21.15.103:0/2234204883 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec12ed20 con 0x7fadec12f250 2023-07-02T09:04:48.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.177+0000 7fadeb7fe700 1 --2- 172.21.15.103:0/2234204883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec12f250 0x7fadec12f660 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d1bb748f902373 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:48.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadea7fc700 1 -- 172.21.15.103:0/2234204883 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fadd400ac60 con 0x7fadec12f250 2023-07-02T09:04:48.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadea7fc700 1 -- 172.21.15.103:0/2234204883 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fadd400adc0 con 0x7fadec12f250 2023-07-02T09:04:48.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadea7fc700 1 -- 172.21.15.103:0/2234204883 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fadd400f470 con 0x7fadec12f250 2023-07-02T09:04:48.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadf20c1700 1 -- 172.21.15.103:0/2234204883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec12f250 msgr2=0x7fadec12f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:48.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadf20c1700 1 --2- 172.21.15.103:0/2234204883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec12f250 0x7fadec12f660 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:48.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadf20c1700 1 -- 172.21.15.103:0/2234204883 shutdown_connections 2023-07-02T09:04:48.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadf20c1700 1 --2- 172.21.15.103:0/2234204883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec12f250 0x7fadec12f660 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:48.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadf20c1700 1 -- 172.21.15.103:0/2234204883 >> 172.21.15.103:0/2234204883 conn(0x7fadec09d810 msgr2=0x7fadec09dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:48.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadf20c1700 1 -- 172.21.15.103:0/2234204883 shutdown_connections 2023-07-02T09:04:48.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.178+0000 7fadf20c1700 1 -- 172.21.15.103:0/2234204883 wait complete. 2023-07-02T09:04:48.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.179+0000 7fadf20c1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:48.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.179+0000 7fadf20c1700 1 Processor -- start 2023-07-02T09:04:48.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.180+0000 7fadf20c1700 1 -- start start 2023-07-02T09:04:48.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.180+0000 7fadf20c1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec098a00 0x7fadec117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:48.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.180+0000 7fadf20c1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec11a9f0 con 0x7fadec098a00 2023-07-02T09:04:48.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.180+0000 7fadeb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec098a00 0x7fadec117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:48.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.180+0000 7fadeb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec098a00 0x7fadec117360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41370/0 (socket says 172.21.15.103:41370) 2023-07-02T09:04:48.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.180+0000 7fadeb7fe700 1 -- 172.21.15.103:0/2581523489 learned_addr learned my addr 172.21.15.103:0/2581523489 (peer_addr_for_me v2:172.21.15.103:0/0) 2023-07-02T09:04:48.183 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.181+0000 7fadeb7fe700 1 -- 172.21.15.103:0/2581523489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec118f50 con 0x7fadec098a00 2023-07-02T09:04:48.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.181+0000 7fadeb7fe700 1 --2- 172.21.15.103:0/2581523489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec098a00 0x7fadec117360 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:48.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.181+0000 7fade3fff700 1 -- 172.21.15.103:0/2581523489 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fadd4017040 con 0x7fadec098a00 2023-07-02T09:04:48.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.181+0000 7fade3fff700 1 -- 172.21.15.103:0/2581523489 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fadd400fe70 con 0x7fadec098a00 2023-07-02T09:04:48.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.181+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadec11a650 con 0x7fadec098a00 2023-07-02T09:04:48.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.181+0000 7fade3fff700 1 -- 172.21.15.103:0/2581523489 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fadd4011550 con 0x7fadec098a00 2023-07-02T09:04:48.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.181+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadec11afe0 con 0x7fadec098a00 2023-07-02T09:04:48.184 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.182+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadec11a840 con 0x7fadec098a00 2023-07-02T09:04:48.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.182+0000 7fade3fff700 1 -- 172.21.15.103:0/2581523489 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 44759+0+0 (crc 0 0 0) 0x7fadd4011700 con 0x7fadec098a00 2023-07-02T09:04:48.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.182+0000 7fade3fff700 1 --2- 172.21.15.103:0/2581523489 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fadcc0377c0 0x7fadcc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:48.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.182+0000 7fade3fff700 1 -- 172.21.15.103:0/2581523489 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2672+0+0 (crc 0 0 0) 0x7fadd4048c30 con 0x7fadec098a00 2023-07-02T09:04:48.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.183+0000 7fadeaffd700 1 --2- 172.21.15.103:0/2581523489 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fadcc0377c0 0x7fadcc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-02T09:04:48.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.183+0000 7fadeaffd700 1 --2- 172.21.15.103:0/2581523489 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fadcc0377c0 0x7fadcc039cd0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-02T09:04:48.187 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.187+0000 7fade3fff700 1 -- 172.21.15.103:0/2581523489 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (crc 0 0 0) 0x7fadd40103b0 con 0x7fadec098a00 2023-07-02T09:04:48.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.375+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fadec1198a0 con 0x7fadec098a00 2023-07-02T09:04:48.375 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.375+0000 7fade3fff700 1 -- 172.21.15.103:0/2581523489 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+2 (crc 0 0 0) 0x7fadd400c250 con 0x7fadec098a00 2023-07-02T09:04:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-07-02T09:04:48.377 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fadcc0377c0 msgr2=0x7fadcc039cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 --2- 172.21.15.103:0/2581523489 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fadcc0377c0 0x7fadcc039cd0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec098a00 msgr2=0x7fadec117360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-02T09:04:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 --2- 172.21.15.103:0/2581523489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec098a00 0x7fadec117360 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 shutdown_connections 2023-07-02T09:04:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 --2- 172.21.15.103:0/2581523489 >> [v2:172.21.15.164:6808/47615948,v1:172.21.15.164:6809/47615948] conn(0x7fadcc0377c0 0x7fadcc039cd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 --2- 172.21.15.103:0/2581523489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec098a00 0x7fadec117360 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:48.378 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 >> 172.21.15.103:0/2581523489 conn(0x7fadec09d810 msgr2=0x7fadec09f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-02T09:04:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 shutdown_connections 2023-07-02T09:04:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:48.377+0000 7fadf20c1700 1 -- 172.21.15.103:0/2581523489 wait complete. 2023-07-02T09:04:48.390 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2023-07-02T09:04:48.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s 2023-07-02T09:04:48.577 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-07-02T09:04:49.079 INFO:tasks.ceph.osd.0:Sent signal 15 2023-07-02T09:04:49.080 INFO:tasks.ceph.osd.2:Sent signal 15 2023-07-02T09:04:49.080 INFO:tasks.ceph.osd.3:Sent signal 15 2023-07-02T09:04:49.080 INFO:tasks.ceph.mon.a:Sent signal 15 2023-07-02T09:04:49.081 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-07-02T09:04:49.081 INFO:tasks.ceph.mon.a.smithi103.stderr:2023-07-02T09:04:49.081+0000 7f4346425700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 108930) UID: 0 2023-07-02T09:04:49.081 INFO:tasks.ceph.mon.a.smithi103.stderr:2023-07-02T09:04:49.081+0000 7f4346425700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-02T09:04:49.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T09:04:49.121+0000 7f5b89b62700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 108400) UID: 0 2023-07-02T09:04:49.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-07-02T09:04:49.121+0000 7f5b89b62700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-02T09:04:49.392 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T09:04:49.531 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.526+0000 7fe111ac0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:04:49.531 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.531+0000 7fe111ac0700 1 Processor -- start 2023-07-02T09:04:49.532 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.531+0000 7fe111ac0700 1 -- start start 2023-07-02T09:04:49.532 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.532+0000 7fe111ac0700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe10c12f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:49.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.532+0000 7fe111ac0700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe10c098a00 2023-07-02T09:04:49.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.532+0000 7fe111ac0700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe10c12f250 2023-07-02T09:04:49.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.532+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe10c098e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:49.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.532+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe10c12f660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:49.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.532+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe10c12f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:04:49.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.733+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe10c12f660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:49.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.733+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe10c098e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:49.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:49.733+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe10c12f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:04:50.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:50.134+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe10c12f660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:50.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:50.134+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe10c098e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:50.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:50.134+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe10c12f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:04:50.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:50.935+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe10c12f660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:50.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:50.935+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe10c098e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:50.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:50.935+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe10c12f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:04:52.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe10c12f660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:04:52.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe10c12f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:52.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe10c098e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:04:52.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:52.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe10c098a00 2023-07-02T09:04:52.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe10c12f250 2023-07-02T09:04:52.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:52.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe0ec003c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:52.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.532+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:04:52.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.733+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe0ec003c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:52.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.733+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:52.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:52.733+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:04:53.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:53.134+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:53.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:53.134+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe0ec003c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:53.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:53.134+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:04:53.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:53.935+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:53.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:53.935+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe0ec003c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:53.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:53.935+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:04:55.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:04:55.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:55.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe10c098a00 legacy=0x7fe0ec003c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:04:55.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:55.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec00b820 2023-07-02T09:04:55.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe10c12f250 2023-07-02T09:04:55.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:55.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:55.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.533+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:04:55.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.734+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:55.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.734+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:55.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:55.734+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:04:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:56.135+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:56.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:56.135+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:56.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:56.135+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:04:56.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:56.936+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:56.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:56.936+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:56.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:56.936+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:04:58.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 msgr2=0x7fe0ec003ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:04:58.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe10c12f250 0x7fe0ec003ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:04:58.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:04:58.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:04:58.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec00b820 2023-07-02T09:04:58.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec0036f0 2023-07-02T09:04:58.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:58.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:58.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.533+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:04:58.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.734+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:58.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.734+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:58.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:58.734+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:04:59.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:59.135+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:59.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:59.135+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:59.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:59.135+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:04:59.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:59.936+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:04:59.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:59.936+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:04:59.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:04:59.936+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:01.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:01.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:01.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b820 legacy=0x7fe0ec00dc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:01.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:01.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec003cc0 2023-07-02T09:05:01.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec0036f0 2023-07-02T09:05:01.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec00e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:01.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:01.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.533+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:01.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.734+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec00e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:01.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.734+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:01.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:01.734+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:02.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:02.135+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec00e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:02.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:02.135+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:02.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:02.135+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:02.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:02.936+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:02.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:02.936+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec00e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:02.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:02.936+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:04.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.532+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 msgr2=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:04.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.532+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec0036f0 0x7fe0ec00e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:04.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.532+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec00e930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:04.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.532+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:04.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.532+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec003cc0 2023-07-02T09:05:04.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.532+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec006810 2023-07-02T09:05:04.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.532+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:04.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.532+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:04.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.533+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:04.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.734+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:04.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.734+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:04.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:04.734+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:05.135 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:05.135+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:05.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:05.135+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:05.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:05.135+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:05.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:05.936+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:05.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:05.936+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:05.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:05.936+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:07.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:07.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:07.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:07.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:07.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec003cc0 2023-07-02T09:05:07.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec006810 2023-07-02T09:05:07.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:07.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:07.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.533+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:07.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.734+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:07.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.734+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:07.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:07.734+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:08.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:08.135+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:08.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:08.135+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:08.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:08.135+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:08.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:08.936+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:08.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:08.936+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:08.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:08.936+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:10.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 msgr2=0x7fe0ec00dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:10.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec006810 0x7fe0ec00dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:10.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec003cc0 legacy=0x7fe0ec008cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:10.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:10.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec00ae70 2023-07-02T09:05:10.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00a3d0 2023-07-02T09:05:10.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00ae70 legacy=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:10.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:10.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.534+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:10.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.735+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:10.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.735+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00ae70 legacy=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:10.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:10.735+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:11.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:11.136+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00ae70 legacy=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:11.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:11.136+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:11.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:11.136+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:11.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:11.937+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:11.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:11.937+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00ae70 legacy=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:11.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:11.937+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:13.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:13.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:13.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00ae70 legacy=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:13.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:13.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec00dd40 2023-07-02T09:05:13.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00a3d0 2023-07-02T09:05:13.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec010400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:13.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:13.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.534+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:13.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.735+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec010400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:13.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.735+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:13.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:13.735+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:14.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:14.136+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec010400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:14.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:14.136+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:14.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:14.136+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:14.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:14.937+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec010400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:14.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:14.937+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:14.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:14.937+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:16.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 msgr2=0x7fe0ec00a7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:16.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a3d0 0x7fe0ec00a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:16.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec010400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:16.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:16.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec00dd40 2023-07-02T09:05:16.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b340 2023-07-02T09:05:16.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:16.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:16.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.534+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:16.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.735+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:16.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.735+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:16.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:16.735+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:17.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:17.136+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:17.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:17.136+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:17.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:17.136+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:17.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:17.937+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:17.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:17.937+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:17.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:17.937+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:19.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:19.535 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:19.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:19.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:19.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec00a7e0 2023-07-02T09:05:19.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b340 2023-07-02T09:05:19.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:19.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:19.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.535+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:19.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.736+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:19.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.736+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:19.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:19.736+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:20.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:20.137+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:20.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:20.137+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:20.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:20.137+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:20.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:20.938+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0088a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:20.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:20.938+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:20.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:20.938+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:22.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 msgr2=0x7fe0ec006c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:22.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b340 0x7fe0ec006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:22.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0088a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:22.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec007180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:22.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec00a7e0 2023-07-02T09:05:22.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00dd40 2023-07-02T09:05:22.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:22.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0076c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:22.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.535+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec007180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:22.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.736+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:22.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.736+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0076c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:22.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:22.736+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec007180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:23.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:23.137+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:23.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:23.137+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0076c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:23.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:23.137+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec007180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:23.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:23.938+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:23.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:23.938+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0076c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:23.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:23.938+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec007180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:25.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec007180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:25.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec007180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:25.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00a7e0 legacy=0x7fe0ec0076c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:25.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0094f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:25.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec00b320 2023-07-02T09:05:25.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00dd40 2023-07-02T09:05:25.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:25.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0094f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:25.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.535+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:25.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.736+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0094f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:25.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.736+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:25.736 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:25.736+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:26.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:26.137+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:26.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:26.137+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0094f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:26.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:26.137+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:26.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:26.938+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0094f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:26.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:26.938+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:26.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:26.938+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:28.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0094f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:28.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:28.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:28.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 0x7fe0ec009a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:28.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec00b320 2023-07-02T09:05:28.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00a7e0 2023-07-02T09:05:28.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 msgr2=0x7fe0ec009a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:28.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:28.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.536+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 0x7fe0ec009a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:28.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.737+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 msgr2=0x7fe0ec009a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:28.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.737+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:28.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:28.737+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 0x7fe0ec009a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:29.138+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 msgr2=0x7fe0ec009a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:29.138+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:29.138+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 0x7fe0ec009a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:29.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:29.939+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 msgr2=0x7fe0ec009a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:29.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:29.939+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:29.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:29.939+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 0x7fe0ec009a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:31.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 msgr2=0x7fe0ec009a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:31.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00a7e0 0x7fe0ec009a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:31.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00b320 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:31.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 0x7fe0ec009410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:31.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec00dd40 2023-07-02T09:05:31.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec009f70 2023-07-02T09:05:31.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:31.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:31.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.536+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:31.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.737+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:31.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.737+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:31.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:31.737+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:32.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:32.138+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:32.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:32.138+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:32.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:32.138+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:32.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:32.939+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:32.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:32.939+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:32.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:32.939+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:34.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:34.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009f70 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:34.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:34.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:34.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec00dd40 2023-07-02T09:05:34.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b320 2023-07-02T09:05:34.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:34.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:34.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.536+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:34.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.737+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:34.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.737+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:34.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:34.737+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:35.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:35.138+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:35.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:35.138+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:35.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:35.138+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:35.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:35.939+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:35.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:35.939+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:35.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:35.939+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:37.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:37.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:37.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00dd40 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:37.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:37.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec0099f0 2023-07-02T09:05:37.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b320 2023-07-02T09:05:37.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:37.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:37.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.537+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:37.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.738+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:37.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.738+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:37.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:37.738+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:38.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:38.139+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:38.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:38.139+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:38.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:38.139+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:38.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:38.940+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:38.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:38.940+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:38.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:38.940+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:40.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:40.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:40.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec011410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:40.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:40.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec0099f0 2023-07-02T09:05:40.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b320 2023-07-02T09:05:40.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:40.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:40.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.537+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:40.737 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.738+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:40.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.738+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:40.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:40.738+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:41.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:41.139+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:41.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:41.139+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:41.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:41.139+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:41.940+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:41.940+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:41.940+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:43.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:43.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:43.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:43.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:43.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec0099f0 2023-07-02T09:05:43.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b320 2023-07-02T09:05:43.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:43.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:43.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.537+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:43.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.738+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:43.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.738+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:43.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:43.738+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:44.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:44.139+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:44.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:44.139+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:44.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:44.139+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:44.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:44.940+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:44.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:44.940+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:44.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:44.940+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:46.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:46.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:46.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:46.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:46.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec0099f0 2023-07-02T09:05:46.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b320 2023-07-02T09:05:46.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:46.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:46.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.538+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:46.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.739+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:46.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.739+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:46.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:46.739+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:47.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:47.140+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:47.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:47.140+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:47.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:47.140+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:47.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:47.941+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:47.941+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:47.941+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:49.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:49.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:49.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:49.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:49.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec0099f0 2023-07-02T09:05:49.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b320 2023-07-02T09:05:49.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:49.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:49.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.538+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:49.738 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.739+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:49.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.739+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:49.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:49.739+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:50.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:50.140+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:50.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:50.140+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:50.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:50.140+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:50.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:50.941+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:50.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:50.941+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:50.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:50.941+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:52.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:52.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:52.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:52.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:52.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec0099f0 2023-07-02T09:05:52.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00b320 2023-07-02T09:05:52.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:52.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:52.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.538+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:52.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.739+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:52.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.739+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:52.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:52.739+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:53.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:53.140+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:53.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:53.140+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:53.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:53.140+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:53.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:53.940+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:53.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:53.940+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:53.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:53.940+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:55.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 msgr2=0x7fe0ec009e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:55.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00b320 0x7fe0ec009e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:55.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec008c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:55.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0093b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:55.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec0099f0 2023-07-02T09:05:55.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00dd40 2023-07-02T09:05:55.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:55.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:55.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.538+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0093b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:55.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.739+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:55.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.739+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:55.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:55.739+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0093b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:56.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:56.140+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:56.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:56.140+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:56.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:56.140+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0093b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:56.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:56.941+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:56.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:56.941+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:56.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:56.941+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0093b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:05:58.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 msgr2=0x7fe0ec0093b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:58.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00dd40 0x7fe0ec0093b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:05:58.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:05:58.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:05:58.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec0099f0 2023-07-02T09:05:58.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec009e00 2023-07-02T09:05:58.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:58.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:58.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.538+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:05:58.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.739+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:58.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.739+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:58.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:58.739+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:05:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:59.140+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:59.140+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:59.140+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:05:59.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:59.941+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:05:59.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:59.941+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:05:59.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:05:59.941+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:01.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:01.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:01.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:01.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:01.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec0099f0 2023-07-02T09:06:01.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec009e00 2023-07-02T09:06:01.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:01.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:01.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.538+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:01.739 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.739+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:01.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.739+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:01.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:01.739+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:02.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:02.140+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:02.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:02.140+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:02.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:02.140+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:02.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:02.941+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:02.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:02.941+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:02.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:02.941+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:04.539 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 msgr2=0x7fe0ec00eb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:04.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009e00 0x7fe0ec00eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:04.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:04.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 0x7fe0ec009070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:04.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec0099f0 2023-07-02T09:06:04.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec008c60 2023-07-02T09:06:04.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:04.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:04.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.539+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:04.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.740+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:04.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.740+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:04.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:04.740+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:05.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:05.141+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:05.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:05.141+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:05.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:05.141+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:05.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:05.942+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:05.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:05.942+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:05.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:05.942+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:07.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:07.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec008c60 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:07.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099f0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:07.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec012060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:07.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec0125a0 2023-07-02T09:06:07.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00fb10 2023-07-02T09:06:07.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec012060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:07.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec0149c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:07.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.539+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec012060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:07.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.740+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec012060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:07.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.740+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec0149c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:07.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:07.740+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec012060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:08.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:08.141+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec0149c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:08.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:08.141+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec012060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:08.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:08.141+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec012060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:08.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:08.942+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec0149c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:08.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:08.942+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec012060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:08.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:08.942+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec012060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:10.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec012060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:10.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec012060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:10.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec0149c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:10.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec0125a0 2023-07-02T09:06:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00fb10 2023-07-02T09:06:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.539+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:10.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.740+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:10.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.740+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:10.740 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:10.740+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:11.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:11.141+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:11.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:11.141+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:11.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:11.141+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:11.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:11.942+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:11.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:11.942+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:11.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:11.942+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:13.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:13.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:13.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0125a0 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:13.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:13.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec0099d0 2023-07-02T09:06:13.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00fb10 2023-07-02T09:06:13.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099d0 legacy=0x7fe0ec0129b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:13.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:13.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.540+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:13.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.741+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099d0 legacy=0x7fe0ec0129b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:13.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.741+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:13.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:13.741+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:14.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:14.142+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099d0 legacy=0x7fe0ec0129b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:14.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:14.142+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:14.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:14.142+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:14.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:14.943+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099d0 legacy=0x7fe0ec0129b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:14.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:14.943+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:14.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:14.943+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:16.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:16.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:16.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec0099d0 legacy=0x7fe0ec0129b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:16.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:16.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec012060 2023-07-02T09:06:16.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00fb10 2023-07-02T09:06:16.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:16.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:16.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.540+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:16.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.741+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:16.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.741+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:16.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:16.741+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:17.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:17.142+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:17.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:17.142+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:17.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:17.142+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:17.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:17.943+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:17.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:17.943+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:17.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:17.943+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:19.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:19.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:19.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:19.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:19.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec012060 2023-07-02T09:06:19.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00bc60 2023-07-02T09:06:19.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:19.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:19.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.540+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:19.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.741+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:19.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.741+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:19.741 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:19.741+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:20.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:20.142+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:20.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:20.142+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:20.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:20.142+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:20.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:20.943+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:20.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:20.943+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:20.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:20.943+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:22.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:22.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:22.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:22.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec012060 2023-07-02T09:06:22.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00bc60 2023-07-02T09:06:22.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:22.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:22.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.541+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:22.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.742+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:22.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.742+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:22.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:22.742+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:23.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:23.143+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:23.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:23.143+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:23.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:23.143+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:23.944+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:23.944+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:23.944+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:25.541 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 msgr2=0x7fe0ec00c070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:25.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00bc60 0x7fe0ec00c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:25.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec012060 legacy=0x7fe0ec00c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:25.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:25.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec009070 2023-07-02T09:06:25.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00fb10 2023-07-02T09:06:25.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:25.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00ccc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:25.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.541+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:25.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.742+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:25.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.742+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00ccc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:25.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:25.742+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:26.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:26.143+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:26.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:26.143+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00ccc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:26.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:26.143+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:26.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:26.944+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:26.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:26.944+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00ccc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:26.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:26.944+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:28.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00ccc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:28.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:28.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:28.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:28.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec00d200 2023-07-02T09:06:28.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00fb10 2023-07-02T09:06:28.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00d200 legacy=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:28.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00d610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:28.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.541+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00d610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:28.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.742+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00d200 legacy=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:28.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.742+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00d610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:28.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:28.742+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00d610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:29.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:29.143+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00d200 legacy=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:29.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:29.143+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00d610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:29.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:29.143+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00d610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:29.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:29.944+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00d200 legacy=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:29.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:29.944+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00d610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:29.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:29.944+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00d610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:31.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec00d610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:31.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec00d610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:31.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00d200 legacy=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:31.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:31.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec009070 2023-07-02T09:06:31.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00fb10 2023-07-02T09:06:31.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:31.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:31.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.541+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:31.742 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.742+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:31.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.742+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:31.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:31.742+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:32.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:32.143+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:32.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:32.143+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:32.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:32.143+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:32.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:32.944+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:32.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:32.944+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:32.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:32.944+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:34.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 msgr2=0x7fe0ec009480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:34.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00fb10 0x7fe0ec009480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:34.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:34.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:34.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec009070 2023-07-02T09:06:34.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec010030 2023-07-02T09:06:34.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:34.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:34.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.542+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:34.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.743+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:34.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.743+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:34.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:34.743+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:35.144+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:35.144+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:35.144+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:35.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:35.945+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:35.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:35.945+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:35.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:35.945+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:37.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:37.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:37.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009070 legacy=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:37.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:37.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec009480 2023-07-02T09:06:37.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec010030 2023-07-02T09:06:37.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:37.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:37.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.542+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:37.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.743+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:37.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.743+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:37.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:37.743+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:38.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:38.144+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:38.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:38.144+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:38.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:38.144+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:38.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:38.945+0000 7fe10b7fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:38.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:38.945+0000 7fe10affd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:38.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:38.945+0000 7fe10affd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:40.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:40.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec010030 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:40.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:40.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:40.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec009480 2023-07-02T09:06:40.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec009070 2023-07-02T09:06:40.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:40.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:40.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.542+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:40.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.743+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:40.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.743+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:40.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:40.743+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:41.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:41.144+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:41.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:41.144+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:41.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:41.144+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:41.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:41.945+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:41.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:41.945+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:41.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:41.945+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:43.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:43.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00c780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06df60 con 0x7fe0ec009480 2023-07-02T09:06:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec009070 2023-07-02T09:06:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:43.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.543+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:43.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.744+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:43.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.744+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:43.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:43.744+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:44.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:44.145+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:44.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:44.145+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:44.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:44.145+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:44.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:44.946+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:44.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:44.946+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:44.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:44.946+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:46.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe1097fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 msgr2=0x7fe0ec00c240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:46.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec009070 0x7fe0ec00c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:46.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe1097fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec009480 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:46.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe1097fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 0x7fe0ec00bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:46.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe1097fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c12ed20 con 0x7fe0ec00eb00 2023-07-02T09:06:46.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe1097fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c09a710 con 0x7fe0ec00d790 2023-07-02T09:06:46.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00eb00 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:46.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:46.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.543+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:46.743 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.744+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00eb00 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:46.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.744+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:46.744 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:46.744+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:47.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:47.144+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:47.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:47.144+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00eb00 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:47.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:47.144+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:47.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:47.945+0000 7fe10affd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fe0ec00eb00 legacy=0x7fe0ec00e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:47.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:47.945+0000 7fe10b7fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 msgr2=0x7fe0ec00bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:47.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:47.945+0000 7fe10b7fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0ec00d790 0x7fe0ec00bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-07-02T09:06:49.431 INFO:tasks.ceph:ignoring flush pg stats error, probably testing upgrade: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1' 2023-07-02T09:06:49.432 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-07-02T09:06:49.432 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T09:06:49.567 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.563+0000 7fcd89fb2700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:06:49.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.567+0000 7fcd89fb2700 1 Processor -- start 2023-07-02T09:06:49.568 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.567+0000 7fcd89fb2700 1 -- start start 2023-07-02T09:06:49.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.568+0000 7fcd89fb2700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd84133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:49.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.568+0000 7fcd89fb2700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd8409c580 2023-07-02T09:06:49.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.568+0000 7fcd89fb2700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd84132f50 2023-07-02T09:06:49.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.568+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd84133360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:49.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.568+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd8409c580 legacy=0x7fcd8409c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:49.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.568+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd84133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:49.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.769+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd84133360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:49.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.769+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd8409c580 legacy=0x7fcd8409c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:49.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:49.769+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd84133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:50.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:50.170+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd84133360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:50.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:50.170+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd8409c580 legacy=0x7fcd8409c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:50.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:50.170+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd84133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:50.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:50.971+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd84133360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:50.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:50.971+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd8409c580 legacy=0x7fcd8409c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:50.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:50.971+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd84133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:52.569 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd84133360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:52.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd84133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:52.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd8409c580 legacy=0x7fcd8409c990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:52.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd64009350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:52.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64003650 2023-07-02T09:06:52.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd84132f50 2023-07-02T09:06:52.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:52.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:52.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.568+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:52.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.769+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:52.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.769+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:52.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:52.769+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:53.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:53.170+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:53.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:53.170+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:53.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:53.170+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:53.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:53.971+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:53.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:53.971+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:53.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:53.971+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:55.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:55.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd84132f50 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:55.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:55.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:55.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64003650 2023-07-02T09:06:55.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd640099a0 2023-07-02T09:06:55.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:55.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:55.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.569+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:55.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.770+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:55.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.770+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:55.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:55.770+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:56.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:56.171+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:56.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:56.171+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:56.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:56.171+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:56.972 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:56.972+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:56.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:56.972+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:56.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:56.972+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:06:58.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:58.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:06:58.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003650 legacy=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:06:58.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:06:58.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd6400a540 2023-07-02T09:06:58.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd640099a0 2023-07-02T09:06:58.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64006930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:58.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:58.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.569+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:06:58.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.770+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:58.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.770+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64006930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:58.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:58.770+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:06:59.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:59.171+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64006930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:59.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:59.171+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:59.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:59.171+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:06:59.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:59.972+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64006930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:06:59.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:59.972+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:06:59.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:06:59.972+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:01.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:01.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640099a0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:01.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64006930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:01.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64006220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:01.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd6400a540 2023-07-02T09:07:01.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64009db0 2023-07-02T09:07:01.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:01.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:01.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.569+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64006220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:01.770 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.770+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:01.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.770+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:01.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:01.770+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64006220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:02.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:02.171+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:02.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:02.171+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:02.171 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:02.171+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64006220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:02.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:02.972+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:02.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:02.972+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:02.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:02.972+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64006220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:04.570 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64006220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:04.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64006220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:04.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400a540 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:04.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:04.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64006220 2023-07-02T09:07:04.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64009db0 2023-07-02T09:07:04.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:04.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006220 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:04.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.570+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:04.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.771+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:04.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.771+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006220 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:04.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:04.771+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:05.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:05.172+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006220 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:05.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:05.172+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:05.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:05.172+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:05.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:05.973+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006220 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:05.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:05.973+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:05.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:05.973+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:07.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:07.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:07.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006220 legacy=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:07.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:07.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64006630 2023-07-02T09:07:07.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64009db0 2023-07-02T09:07:07.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:07.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:07.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.570+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:07.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.771+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:07.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.771+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:07.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:07.771+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:08.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:08.172+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:08.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:08.172+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:08.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:08.172+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:08.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:08.973+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:08.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:08.973+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:08.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:08.973+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:10.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 msgr2=0x7fcd64009350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:10.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64009db0 0x7fcd64009350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:10.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:10.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:10.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64006630 2023-07-02T09:07:10.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64006ae0 2023-07-02T09:07:10.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:10.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:10.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.570+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:10.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.771+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:10.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.771+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:10.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:10.771+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:11.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:11.172+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:11.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:11.172+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:11.172 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:11.172+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:11.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:11.973+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:11.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:11.973+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:11.973 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:11.973+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:13.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64009330 2023-07-02T09:07:13.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64006ae0 2023-07-02T09:07:13.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:13.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64009330 legacy=0x7fcd6400e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:13.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.571+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:13.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.772+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64009330 legacy=0x7fcd6400e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:13.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.772+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:13.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:13.772+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:14.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:14.173+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64009330 legacy=0x7fcd6400e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:14.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:14.173+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:14.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:14.173+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:14.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:14.974+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64009330 legacy=0x7fcd6400e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:14.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:14.974+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:14.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:14.974+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:16.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:16.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:16.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64009330 legacy=0x7fcd6400e1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:16.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:16.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64006630 2023-07-02T09:07:16.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64006ae0 2023-07-02T09:07:16.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:16.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:16.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.571+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:16.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.772+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:16.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.772+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:16.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:16.772+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:17.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:17.173+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:17.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:17.173+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:17.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:17.173+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:17.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:17.974+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:17.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:17.974+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:17.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:17.974+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:19.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 msgr2=0x7fcd64005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:19.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64006ae0 0x7fcd64005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:19.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:19.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:19.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64006630 2023-07-02T09:07:19.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400c1d0 2023-07-02T09:07:19.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:19.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:19.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.571+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:19.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.772+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:19.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.772+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:19.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:19.772+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:20.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:20.173+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:20.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:20.173+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:20.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:20.173+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:20.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:20.974+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:20.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:20.974+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:20.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:20.974+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:22.571 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:22.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:22.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:22.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:22.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64006a40 2023-07-02T09:07:22.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400c1d0 2023-07-02T09:07:22.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006a40 legacy=0x7fcd6400ebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:22.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:22.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.571+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.772+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006a40 legacy=0x7fcd6400ebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.772+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:22.772+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:23.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:23.173+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006a40 legacy=0x7fcd6400ebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:23.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:23.173+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:23.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:23.173+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:23.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:23.974+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006a40 legacy=0x7fcd6400ebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:23.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:23.974+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:23.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:23.974+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:25.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:25.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:25.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006a40 legacy=0x7fcd6400ebd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:25.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:25.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64006630 2023-07-02T09:07:25.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400c1d0 2023-07-02T09:07:25.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:25.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:25.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.572+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:25.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.773+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:25.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.773+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:25.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:25.773+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:26.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:26.174+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:26.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:26.174+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:26.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:26.174+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:26.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:26.975+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:26.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:26.975+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:26.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:26.975+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:28.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:28.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400c1d0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:28.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:28.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 0x7fcd6400e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:28.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64006630 2023-07-02T09:07:28.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64005be0 2023-07-02T09:07:28.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:28.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:28.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.572+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:28.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.773+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:28.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.773+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:28.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:28.773+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:29.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:29.174+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:29.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:29.174+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:29.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:29.174+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:29.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:29.975+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:29.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:29.975+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:29.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:29.975+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 msgr2=0x7fcd6400e5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005be0 0x7fcd6400e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd6400dd70 2023-07-02T09:07:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400b1c0 2023-07-02T09:07:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400dd70 legacy=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.572+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:31.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.773+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400dd70 legacy=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:31.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.773+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:31.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:31.773+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:32.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:32.174+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:32.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:32.174+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400dd70 legacy=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:32.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:32.174+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:32.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:32.975+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:32.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:32.975+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400dd70 legacy=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:32.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:32.975+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:34.572 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400d830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:34.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:34.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400dd70 legacy=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:34.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd640051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:34.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64006630 2023-07-02T09:07:34.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400b1c0 2023-07-02T09:07:34.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd640051e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:34.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:34.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.572+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd640051e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:34.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.773+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:34.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.773+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd640051e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:34.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:34.773+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd640051e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:35.173 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:35.174+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:35.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:35.174+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd640051e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:35.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:35.174+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd640051e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:35.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:35.975+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:35.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:35.975+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd640051e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:35.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:35.975+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd640051e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:37.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd640051e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:37.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd640051e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:37.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64006630 legacy=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:37.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:37.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64004ad0 2023-07-02T09:07:37.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400b1c0 2023-07-02T09:07:37.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:37.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:37.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.573+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:37.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.774+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:37.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.774+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:37.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:37.774+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:38.174 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:38.175+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:38.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:38.175+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:38.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:38.175+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:38.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:38.976+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:38.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:38.976+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:38.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:38.976+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 msgr2=0x7fcd6400aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400b1c0 0x7fcd6400aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64004ad0 2023-07-02T09:07:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400d830 2023-07-02T09:07:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.572+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400dc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:40.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.773+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:40.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.773+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:40.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:40.773+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400dc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:41.174+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:41.174+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:41.174+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400dc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:41.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:41.975+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400dc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:41.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:41.975+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:41.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:41.975+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400dc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:43.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400dc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:43.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400dc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:43.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004ad0 legacy=0x7fcd6400e220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:43.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:43.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd6400b090 2023-07-02T09:07:43.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400d830 2023-07-02T09:07:43.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400b090 legacy=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:43.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400aa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:43.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.572+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:43.773 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.773+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400b090 legacy=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:43.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.773+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400aa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:43.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:43.773+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:44.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:44.174+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400b090 legacy=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:44.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:44.174+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400aa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:44.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:44.174+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:44.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:44.975+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400b090 legacy=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:44.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:44.975+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400aa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:44.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:44.975+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:46.573 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400aa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:46.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:46.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400b090 legacy=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:46.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:46.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd640055f0 2023-07-02T09:07:46.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64004ad0 2023-07-02T09:07:46.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:46.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640055f0 legacy=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:46.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.573+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:46.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.774+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640055f0 legacy=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:46.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.774+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:46.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:46.774+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:47.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:47.175+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640055f0 legacy=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:47.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:47.175+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:47.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:47.175+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:47.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:47.976+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640055f0 legacy=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:47.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:47.976+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:47.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:47.976+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:49.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:49.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:49.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640055f0 legacy=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:49.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:49.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd6400aa30 2023-07-02T09:07:49.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64004ad0 2023-07-02T09:07:49.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:49.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:49.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.573+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:49.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.774+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:49.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.774+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:49.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:49.774+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:50.175 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:50.175+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:50.175+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:50.175+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:50.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:50.976+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:50.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:50.976+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:50.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:50.976+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:52.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 msgr2=0x7fcd64004ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:52.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64004ad0 0x7fcd64004ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:52.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64010360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:52.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:52.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd6400aa30 2023-07-02T09:07:52.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400d830 2023-07-02T09:07:52.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:52.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:52.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.573+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:52.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.774+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:52.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.774+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:52.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:52.774+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:53.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:53.175+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:53.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:53.175+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:53.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:53.175+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:53.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:53.976+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:53.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:53.976+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:53.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:53.976+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:55.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 msgr2=0x7fcd6400fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:55.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400d830 0x7fcd6400fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:55.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa30 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:55.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 0x7fcd64013cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:55.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd640051e0 2023-07-02T09:07:55.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd640118b0 2023-07-02T09:07:55.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 msgr2=0x7fcd64013cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:55.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 0x7fcd64013cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:55.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.573+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:55.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.774+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:55.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.774+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 msgr2=0x7fcd64013cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:55.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:55.774+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 0x7fcd64013cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:56.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:56.175+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:56.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:56.175+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 msgr2=0x7fcd64013cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:56.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:56.175+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 0x7fcd64013cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:56.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:56.976+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:56.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:56.976+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 msgr2=0x7fcd64013cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:56.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:56.976+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 0x7fcd64013cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:07:58.574 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 msgr2=0x7fcd64013cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:58.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640118b0 0x7fcd64013cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:07:58.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:07:58.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd640108a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:07:58.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd6400aa10 2023-07-02T09:07:58.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400fc50 2023-07-02T09:07:58.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:58.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd640108a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:58.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.574+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd640108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:07:58.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.775+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:58.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.775+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd640108a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:58.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:58.775+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd640108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:07:59.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:59.176+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:59.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:59.176+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd640108a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:59.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:59.176+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd640108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:07:59.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:59.977+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:07:59.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:59.977+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd640108a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:07:59.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:07:59.977+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd640108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:01.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd640108a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:01.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd640108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:01.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:01.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:01.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd640051e0 2023-07-02T09:08:01.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400fc50 2023-07-02T09:08:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.574+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:01.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.775+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:01.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.775+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:01.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:01.775+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:02.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:02.176+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:02.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:02.176+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:02.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:02.176+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:02.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:02.977+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:02.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:02.977+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:02.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:02.977+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:04.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:04.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:04.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640051e0 legacy=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:04.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:04.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd6400aa10 2023-07-02T09:08:04.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400fc50 2023-07-02T09:08:04.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd6400ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:04.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:04.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.574+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:04.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.775+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd6400ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:04.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.775+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:04.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:04.775+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:05.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:05.176+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd6400ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:05.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:05.176+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:05.176 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:05.176+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:05.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:05.977+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd6400ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:05.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:05.977+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:05.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:05.977+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:07.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd6400c8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:07.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd6400c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:07.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd6400ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:07.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 0x7fcd64004ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:07.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd6400aa10 2023-07-02T09:08:07.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64010080 2023-07-02T09:08:07.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 msgr2=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:07.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:07.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.575+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 0x7fcd64004ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:07.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.776+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 msgr2=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:07.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.776+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:07.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:07.776+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 0x7fcd64004ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:08.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:08.177+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 msgr2=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:08.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:08.177+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:08.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:08.177+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 0x7fcd64004ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:08.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:08.978+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 msgr2=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:08.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:08.978+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:08.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:08.978+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 0x7fcd64004ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:10.575 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 msgr2=0x7fcd64004ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:10.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64010080 0x7fcd64004ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:10.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:10.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd64008270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:10.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd6400fc50 2023-07-02T09:08:10.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64005010 2023-07-02T09:08:10.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:10.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd64008270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:10.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.575+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd64008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:10.775 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.776+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:10.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.776+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd64008270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:10.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:10.776+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd64008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:11.177+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:11.177+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd64008270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:11.177+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd64008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:11.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:11.978+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:11.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:11.978+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd64008270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:11.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:11.978+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd64008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:13.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd64008270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:13.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd64008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:13.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:13.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd640041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:13.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64010080 2023-07-02T09:08:13.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64005010 2023-07-02T09:08:13.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:13.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:13.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.575+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:13.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.776+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:13.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.776+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:13.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:13.776+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:14.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:14.177+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:14.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:14.177+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:14.177 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:14.177+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:14.978+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:14.978+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:14.978+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:16.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:16.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64005010 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:16.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:16.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd64004710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:16.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64010080 2023-07-02T09:08:16.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400fc50 2023-07-02T09:08:16.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:16.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64004c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:16.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.576+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd64004710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:16.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.777+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:16.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.777+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64004c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:16.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:16.777+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd64004710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:17.178+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:17.178+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64004c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:17.178+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd64004710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:17.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:17.979+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:17.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:17.979+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64004c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:17.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:17.979+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd64004710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:19.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 msgr2=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:19.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc50 0x7fcd64004710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:19.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64010080 legacy=0x7fcd64004c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:19.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:19.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64004000 2023-07-02T09:08:19.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400aa10 2023-07-02T09:08:19.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:19.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:19.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.576+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:19.776 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.777+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:19.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.777+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:19.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:19.777+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:20.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:20.178+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:20.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:20.178+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:20.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:20.178+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:20.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:20.979+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:20.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:20.979+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:20.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:20.979+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:22.576 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:22.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:22.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:22.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:22.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd6400fc50 2023-07-02T09:08:22.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400aa10 2023-07-02T09:08:22.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:22.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:22.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.576+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:22.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.777+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:22.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.777+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:22.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:22.777+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:23.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:23.178+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:23.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:23.178+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:23.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:23.178+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:23.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:23.979+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:23.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:23.979+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:23.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:23.979+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:25.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:25.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:25.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:25.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:25.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd64004000 2023-07-02T09:08:25.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400aa10 2023-07-02T09:08:25.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:25.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:25.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.577+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:25.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.778+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:25.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.778+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:25.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:25.778+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:26.178 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:26.179+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:26.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:26.179+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:26.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:26.179+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:26.979 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:26.980+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:26.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:26.980+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:26.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:26.980+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:28.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:28.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:28.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64004000 legacy=0x7fcd64004410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:28.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64010080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:28.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd6400fc50 2023-07-02T09:08:28.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400aa10 2023-07-02T09:08:28.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:28.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:28.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.577+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:28.777 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.778+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:28.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.778+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:28.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:28.778+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:29.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:29.179+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:29.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:29.179+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:29.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:29.179+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:29.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:29.980+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:29.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:29.980+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:29.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:29.980+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:31.577 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 msgr2=0x7fcd64010080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:31.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400aa10 0x7fcd64010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:31.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400fc50 legacy=0x7fcd64004b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:31.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd640041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:31.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd640105c0 2023-07-02T09:08:31.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd640109d0 2023-07-02T09:08:31.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:31.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640105c0 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:31.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.578+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:31.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.779+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640105c0 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:31.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.779+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:31.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:31.779+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:32.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:32.180+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640105c0 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:32.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:32.180+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:32.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:32.180+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:32.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:32.981+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640105c0 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:32.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:32.981+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:32.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:32.981+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:34.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd640041d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:34.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd640041d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:34.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640105c0 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:34.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:34.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd6400aa10 2023-07-02T09:08:34.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd640109d0 2023-07-02T09:08:34.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:34.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:34.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.577+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:34.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.778+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:34.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.778+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:34.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:34.778+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:35.179 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:35.179+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:35.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:35.179+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:35.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:35.179+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:35.980 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:35.980+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:35.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:35.980+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:35.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:35.980+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:37.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:37.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:37.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:37.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:37.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd6400aa10 2023-07-02T09:08:37.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd640109d0 2023-07-02T09:08:37.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:37.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:37.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.577+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:37.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.778+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:37.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.778+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:37.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:37.778+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:38.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:38.179+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:38.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:38.179+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:38.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:38.179+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:38.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:38.980+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:38.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:38.980+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:38.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:38.980+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:40.578 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64004e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:40.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64004e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:40.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:40.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:40.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd640041d0 2023-07-02T09:08:40.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd640109d0 2023-07-02T09:08:40.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640041d0 legacy=0x7fcd640045e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:40.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:40.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.578+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:40.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.779+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640041d0 legacy=0x7fcd640045e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:40.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.779+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:40.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:40.779+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:41.180 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:41.180+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640041d0 legacy=0x7fcd640045e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:41.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:41.180+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:41.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:41.180+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:41.981 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:41.981+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640041d0 legacy=0x7fcd640045e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:41.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:41.981+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:41.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:41.981+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:43.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 msgr2=0x7fcd64003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:43.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd640109d0 0x7fcd64003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:43.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd640041d0 legacy=0x7fcd640045e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:43.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 0x7fcd64010070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:43.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8409e260 con 0x7fcd6400aa10 2023-07-02T09:08:43.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd6400fc60 2023-07-02T09:08:43.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 msgr2=0x7fcd64010070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:43.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:43.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.578+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 0x7fcd64010070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:43.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.779+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:43.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.779+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 msgr2=0x7fcd64010070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:43.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:43.779+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 0x7fcd64010070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:44.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:44.180+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:44.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:44.180+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 msgr2=0x7fcd64010070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:44.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:44.180+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 0x7fcd64010070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:44.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:44.981+0000 7fcd837fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:44.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:44.981+0000 7fcd82ffd700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 msgr2=0x7fcd64010070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:44.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:44.981+0000 7fcd82ffd700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 0x7fcd64010070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:46.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd817fa700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 msgr2=0x7fcd64010070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:46.579 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd6400fc60 0x7fcd64010070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:46.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd817fa700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd6400aa10 legacy=0x7fcd64004cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:46.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd817fa700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 0x7fcd640105b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:46.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd817fa700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84132a20 con 0x7fcd64003ed0 2023-07-02T09:08:46.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd817fa700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd840a1640 con 0x7fcd64003ac0 2023-07-02T09:08:46.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003ed0 legacy=0x7fcd640042e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:46.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 msgr2=0x7fcd640105b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:46.580 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.578+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 0x7fcd640105b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:46.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.779+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003ed0 legacy=0x7fcd640042e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:46.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.779+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 msgr2=0x7fcd640105b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:46.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:46.779+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 0x7fcd640105b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:47.180+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003ed0 legacy=0x7fcd640042e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:47.180+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 msgr2=0x7fcd640105b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:47.181 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:47.180+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 0x7fcd640105b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:47.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:47.981+0000 7fcd837fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 msgr2=0x7fcd640105b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:47.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:47.981+0000 7fcd82ffd700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fcd64003ed0 legacy=0x7fcd640042e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:47.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:47.981+0000 7fcd837fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd64003ac0 0x7fcd640105b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-07-02T09:08:49.469 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-07-02T09:08:49.469 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T09:08:49.609 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.604+0000 7f35ae729700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-07-02T09:08:49.609 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.608+0000 7f35ae729700 1 Processor -- start 2023-07-02T09:08:49.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.608+0000 7f35ae729700 1 -- start start 2023-07-02T09:08:49.610 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.609+0000 7f35ae729700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 0x7f35a8133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:49.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.609+0000 7f35ae729700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35a809c580 2023-07-02T09:08:49.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.609+0000 7f35ae729700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f35a8132f50 2023-07-02T09:08:49.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.609+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 msgr2=0x7f35a8133360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:49.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.609+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f35a809c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:49.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.609+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 0x7f35a8133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:49.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.810+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 msgr2=0x7f35a8133360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:49.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.810+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f35a809c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:49.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:49.810+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 0x7f35a8133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:50.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:50.211+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f35a809c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:50.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:50.211+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 msgr2=0x7f35a8133360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:50.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:50.211+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 0x7f35a8133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:51.012 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:51.012+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f35a809c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:51.013 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:51.012+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 msgr2=0x7f35a8133360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:51.013 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:51.012+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 0x7f35a8133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:52.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.610+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 msgr2=0x7f35a8133360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:52.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.610+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35a8132f50 0x7f35a8133360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:52.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.610+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f35a809c990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:52.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.610+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 0x7f3594005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:52.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.610+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35a809c580 2023-07-02T09:08:52.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.610+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594003650 2023-07-02T09:08:52.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.611+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:52.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.611+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:52.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.611+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:52.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.812+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:52.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.812+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:52.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:52.812+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:53.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:53.213+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:53.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:53.213+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:53.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:53.213+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:54.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:54.014+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:54.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:54.014+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:54.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:54.014+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:55.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:55.611 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594003650 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:55.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:55.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 0x7f3594005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:55.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35a809c580 2023-07-02T09:08:55.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594009980 2023-07-02T09:08:55.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:55.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:55.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.611+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:55.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.812+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:55.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.812+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:55.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:55.812+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:56.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:56.213+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:56.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:56.213+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:56.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:56.213+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:08:57.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:57.014+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:57.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:57.014+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:57.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:57.014+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:08:58.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 msgr2=0x7f3594005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:58.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009980 0x7f3594005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:08:58.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:08:58.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 0x7f359400a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:08:58.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35a809c580 2023-07-02T09:08:58.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594009f40 2023-07-02T09:08:58.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 msgr2=0x7f359400a3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:58.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:58.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.611+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 0x7f359400a3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:08:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.812+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 msgr2=0x7f359400a3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.812+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:58.812+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 0x7f359400a3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:08:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:59.213+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 msgr2=0x7f359400a3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:08:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:59.213+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:08:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:08:59.213+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 0x7f359400a3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:00.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:00.014+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:00.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:00.014+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 msgr2=0x7f359400a3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:00.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:00.014+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 0x7f359400a3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:01.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 msgr2=0x7f359400a3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:01.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009f40 0x7f359400a3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:01.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35a809c580 legacy=0x7f3594008880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:01.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 0x7f3594008f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:01.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f359400a350 2023-07-02T09:09:01.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594005b20 2023-07-02T09:09:01.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 msgr2=0x7f3594008f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:01.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400a350 legacy=0x7f3594007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:01.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.611+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 0x7f3594008f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:01.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.812+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 msgr2=0x7f3594008f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:01.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.812+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400a350 legacy=0x7f3594007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:01.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:01.812+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 0x7f3594008f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:02.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:02.213+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400a350 legacy=0x7f3594007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:02.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:02.213+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 msgr2=0x7f3594008f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:02.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:02.213+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 0x7f3594008f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:03.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:03.014+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400a350 legacy=0x7f3594007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:03.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:03.014+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 msgr2=0x7f3594008f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:03.014 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:03.014+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 0x7f3594008f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:04.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 msgr2=0x7f3594008f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:04.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594005b20 0x7f3594008f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:04.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400a350 legacy=0x7f3594007150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:04.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 0x7f3594011bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:04.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594008880 2023-07-02T09:09:04.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007f70 2023-07-02T09:09:04.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008880 legacy=0x7f3594007860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:04.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 msgr2=0x7f3594011bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:04.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.612+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 0x7f3594011bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:04.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.813+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 msgr2=0x7f3594011bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:04.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.813+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008880 legacy=0x7f3594007860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:04.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:04.813+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 0x7f3594011bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:05.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:05.214+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008880 legacy=0x7f3594007860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:05.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:05.214+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 msgr2=0x7f3594011bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:05.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:05.214+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 0x7f3594011bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:06.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:06.015+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008880 legacy=0x7f3594007860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:06.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:06.015+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 msgr2=0x7f3594011bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:06.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:06.015+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 0x7f3594011bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:07.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 msgr2=0x7f3594011bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:07.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007f70 0x7f3594011bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008880 legacy=0x7f3594007860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 0x7f3594007150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594008f20 2023-07-02T09:09:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594008380 2023-07-02T09:09:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 msgr2=0x7f3594007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008f20 legacy=0x7f359400ebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.612+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 0x7f3594007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:07.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.813+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 msgr2=0x7f3594007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:07.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.813+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008f20 legacy=0x7f359400ebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:07.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:07.813+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 0x7f3594007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:08.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:08.214+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008f20 legacy=0x7f359400ebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:08.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:08.214+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 msgr2=0x7f3594007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:08.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:08.214+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 0x7f3594007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:09.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:09.015+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 msgr2=0x7f3594007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:09.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:09.015+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008f20 legacy=0x7f359400ebd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:09.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:09.015+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 0x7f3594007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:10.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 msgr2=0x7f3594007150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594008380 0x7f3594007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594008f20 legacy=0x7f359400ebd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f359400f2e0 2023-07-02T09:09:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594009330 2023-07-02T09:09:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.612+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400f2e0 legacy=0x7f3594011700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:10.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.813+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:10.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.813+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400f2e0 legacy=0x7f3594011700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:10.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:10.813+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:11.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:11.214+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:11.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:11.214+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400f2e0 legacy=0x7f3594011700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:11.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:11.214+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:12.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:12.015+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400f2e0 legacy=0x7f3594011700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:12.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:12.015+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:12.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:12.015+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:13.612 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:13.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:13.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400f2e0 legacy=0x7f3594011700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:13.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:13.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594007150 2023-07-02T09:09:13.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594009330 2023-07-02T09:09:13.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007150 legacy=0x7f3594006120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:13.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:13.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.612+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:13.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.813+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:13.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.813+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007150 legacy=0x7f3594006120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:13.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:13.813+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:14.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:14.214+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007150 legacy=0x7f3594006120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:14.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:14.214+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:14.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:14.214+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:15.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:15.015+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:15.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:15.015+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007150 legacy=0x7f3594006120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:15.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:15.015+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:16.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:16.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594009330 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:16.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007150 legacy=0x7f3594006120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:16.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f359400e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:16.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594006830 2023-07-02T09:09:16.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006c40 2023-07-02T09:09:16.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:16.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006830 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:16.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.613+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f359400e2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:16.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.814+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:16.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.814+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006830 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:16.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:16.814+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f359400e2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:17.215+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:17.215+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006830 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:17.215+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f359400e2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:18.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:18.016+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:18.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:18.016+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006830 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:18.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:18.016+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f359400e2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:19.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.612+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:19.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.612+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f359400e2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:19.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.612+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006830 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:19.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:19.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.613+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594007050 2023-07-02T09:09:19.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.613+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006c40 2023-07-02T09:09:19.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.613+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:19.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.613+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:19.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.613+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.814+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.814+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:19.814+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:20.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:20.215+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:20.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:20.215+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:20.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:20.215+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:21.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:21.016+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:21.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:21.016+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:21.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:21.016+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:22.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:22.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:22.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:22.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:22.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35940067a0 2023-07-02T09:09:22.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006c40 2023-07-02T09:09:22.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:22.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:22.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.613+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.814+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.814+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:22.814+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:23.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:23.215+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:23.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:23.215+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:23.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:23.215+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:24.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:24.016+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:24.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:24.016+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:24.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:24.016+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:25.613 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:25.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:25.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:25.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:25.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35940067a0 2023-07-02T09:09:25.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006c40 2023-07-02T09:09:25.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:25.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:25.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.613+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:25.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.814+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:25.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.814+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:25.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:25.814+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:26.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:26.215+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:26.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:26.215+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:26.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:26.215+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:27.015 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:27.015+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:27.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:27.015+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:27.016 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:27.015+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:28.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:28.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:28.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:28.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:28.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594007050 2023-07-02T09:09:28.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006c40 2023-07-02T09:09:28.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:28.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:28.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.613+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:28.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.814+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:28.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.814+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:28.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:28.814+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:29.215+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:29.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:29.215+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:29.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:29.215+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:30.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:30.016+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:30.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:30.016+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:30.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:30.016+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:31.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:31.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594007050 2023-07-02T09:09:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006c40 2023-07-02T09:09:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.613+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:31.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.814+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:31.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.814+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:31.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:31.814+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:32.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:32.215+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:32.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:32.215+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:32.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:32.215+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:33.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:33.016+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:33.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:33.016+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:33.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:33.016+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:34.614 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:34.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:34.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007050 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:34.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:34.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35940067a0 2023-07-02T09:09:34.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006c40 2023-07-02T09:09:34.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:34.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:34.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.614+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:34.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.815+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:34.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.815+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:34.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:34.815+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:35.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:35.216+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:35.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:35.216+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:35.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:35.216+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:36.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:36.017+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:36.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:36.017+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:36.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:36.017+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:37.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:37.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006c40 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:37.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:37.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:37.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35940067a0 2023-07-02T09:09:37.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007050 2023-07-02T09:09:37.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:37.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:37.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.614+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:37.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.815+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:37.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.815+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:37.815 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:37.815+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:38.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:38.216+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:38.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:38.216+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:38.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:38.216+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:39.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:39.017+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:39.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:39.017+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:39.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:39.017+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:40.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:40.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:40.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:40.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:40.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35940067a0 2023-07-02T09:09:40.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007050 2023-07-02T09:09:40.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:40.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:40.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.614+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:40.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.815+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:40.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.815+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:40.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:40.815+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:41.216 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:41.216+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:41.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:41.216+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:41.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:41.216+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:42.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:42.017+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:42.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:42.017+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:42.017 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:42.017+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:43.615 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:43.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:43.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:43.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:43.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35940067a0 2023-07-02T09:09:43.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007050 2023-07-02T09:09:43.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:43.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:43.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.615+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:43.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.816+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:43.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.816+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:43.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:43.816+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:44.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:44.217+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:44.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:44.217+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:44.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:44.217+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:45.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:45.018+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:45.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:45.018+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:45.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:45.018+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:46.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:46.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:46.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:46.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:46.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35940067a0 2023-07-02T09:09:46.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007050 2023-07-02T09:09:46.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:46.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:46.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.615+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:46.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.816+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:46.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.816+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:46.816 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:46.816+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:47.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:47.217+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:47.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:47.217+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:47.217 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:47.217+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:48.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:48.018+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:48.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:48.018+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:48.018 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:48.018+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:49.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.615+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:49.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.615+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:49.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.615+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:49.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.615+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:49.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.616+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35940067a0 2023-07-02T09:09:49.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.616+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007050 2023-07-02T09:09:49.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.616+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:49.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.616+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:49.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.616+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:49.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.817+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:49.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.817+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:49.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:49.817+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:50.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:50.218+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:50.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:50.218+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:50.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:50.218+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:51.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:51.019+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:51.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:51.019+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:51.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:51.019+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:52.616 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:52.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007050 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:52.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:52.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f359400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:52.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35940067a0 2023-07-02T09:09:52.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:09:52.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:52.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:52.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.616+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:52.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.817+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:52.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.817+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:52.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:52.817+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:53.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:53.218+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:53.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:53.218+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:53.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:53.218+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:54.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:54.019+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:54.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:54.019+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:54.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:54.019+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:55.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:55.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:55.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:55.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:55.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35940067a0 2023-07-02T09:09:55.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:09:55.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:55.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:55.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.616+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:55.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.817+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:55.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.817+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:55.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:55.817+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:56.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:56.218+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:56.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:56.218+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:56.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:56.218+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:09:57.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:57.019+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:57.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:57.019+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:57.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:57.019+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:09:58.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:58.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:09:58.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940067a0 legacy=0x7f3594006090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:09:58.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:09:58.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594006fc0 2023-07-02T09:09:58.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:09:58.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:58.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:58.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.616+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:09:58.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.817+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:58.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.817+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:58.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:58.817+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:09:59.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:59.218+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:09:59.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:59.218+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:09:59.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:09:59.218+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:00.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:00.019+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:00.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:00.019+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:00.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:00.019+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:01.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:01.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:01.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:01.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:01.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594006090 2023-07-02T09:10:01.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:10:01.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:01.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:01.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.617+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:01.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.818+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:01.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.818+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:01.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:01.818+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:02.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:02.219+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:02.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:02.219+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:02.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:02.219+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:03.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:03.020+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:03.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:03.020+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:03.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:03.020+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:04.617 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:04.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:04.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:04.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:04.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594006090 2023-07-02T09:10:04.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:10:04.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:04.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:04.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.617+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:04.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.818+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:04.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.818+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:04.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:04.818+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:05.219+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:05.219+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:05.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:05.219+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:06.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:06.020+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:06.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:06.020+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:06.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:06.020+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 0x7f359400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594006090 2023-07-02T09:10:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006fc0 2023-07-02T09:10:07.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:07.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:07.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.617+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:07.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.818+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:07.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.818+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:07.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:07.818+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:08.218 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:08.219+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:08.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:08.219+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:08.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:08.219+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:09.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:09.020+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:09.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:09.020+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:09.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:09.020+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006fc0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594006090 2023-07-02T09:10:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:10:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.618+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:10.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.819+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:10.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.819+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:10.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:10.819+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:11.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:11.220+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:11.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:11.220+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:11.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:11.220+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:12.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:12.021+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:12.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:12.021+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:12.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:12.021+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:13.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:13.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:13.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006090 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:13.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:13.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594006fc0 2023-07-02T09:10:13.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:10:13.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:13.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:13.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.618+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:13.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.819+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:13.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.819+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:13.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:13.819+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:14.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:14.220+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:14.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:14.220+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:14.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:14.220+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:15.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:15.021+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:15.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:15.021+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:15.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:15.021+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:16.618 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:16.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:16.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:16.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:16.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f359400dbd0 2023-07-02T09:10:16.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:10:16.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400dbd0 legacy=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:16.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:16.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.618+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:16.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.819+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400dbd0 legacy=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:16.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.819+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:16.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:16.819+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:17.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:17.220+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:17.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:17.220+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400dbd0 legacy=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:17.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:17.220+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:18.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:18.021+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400dbd0 legacy=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:18.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:18.021+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:18.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:18.021+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:19.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:19.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:19.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400dbd0 legacy=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:19.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:19.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594006fc0 2023-07-02T09:10:19.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:10:19.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:19.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:19.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.618+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:19.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.819+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:19.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.819+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:19.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:19.819+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:20.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:20.220+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:20.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:20.220+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:20.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:20.220+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:21.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:21.021+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:21.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:21.021+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:21.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:21.021+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:22.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:22.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:22.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:22.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:22.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594006fc0 2023-07-02T09:10:22.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006bb0 2023-07-02T09:10:22.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:22.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:22.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.618+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:22.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.819+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:22.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.819+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:22.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:22.819+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:23.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:23.220+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:23.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:23.220+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:23.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:23.220+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:24.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:24.021+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:24.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:24.021+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:24.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:24.021+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:25.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 msgr2=0x7f3594007730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:25.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006bb0 0x7f3594007730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:25.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006fc0 legacy=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:25.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 0x7f359400dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:25.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f359400e520 2023-07-02T09:10:25.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f359400dbd0 2023-07-02T09:10:25.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400e520 legacy=0x7f359400e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:25.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 msgr2=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:25.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.618+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 0x7f359400dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:25.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.819+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400e520 legacy=0x7f359400e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:25.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.819+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 msgr2=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:25.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:25.819+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 0x7f359400dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:26.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:26.220+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400e520 legacy=0x7f359400e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:26.220 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:26.220+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 msgr2=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:26.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:26.220+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 0x7f359400dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:27.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:27.021+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400e520 legacy=0x7f359400e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:27.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:27.021+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 msgr2=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:27.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:27.021+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 0x7f359400dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:28.619 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 msgr2=0x7f359400dfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:28.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400dbd0 0x7f359400dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:28.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f359400e520 legacy=0x7f359400e930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:28.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:28.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35940069e0 2023-07-02T09:10:28.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007730 2023-07-02T09:10:28.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:28.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:28.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.619+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:28.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.820+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:28.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.820+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:28.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:28.820+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:29.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:29.221+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:29.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:29.221+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:29.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:29.221+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:30.022 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:30.022+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:30.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:30.022+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:30.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:30.022+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:31.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:31.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:31.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:31.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:31.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f3594006df0 2023-07-02T09:10:31.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f359400e9f0 2023-07-02T09:10:31.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:31.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:31.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.618+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:31.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.820+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:31.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.820+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:31.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:31.820+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:32.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:32.221+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:32.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:32.221+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:32.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:32.221+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:33.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:33.022+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:33.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:33.022+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:33.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:33.022+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:34.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:34.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:34.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400e2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:34.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:34.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f35940069e0 2023-07-02T09:10:34.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007730 2023-07-02T09:10:34.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400e9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:34.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:34.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.619+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:34.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.820+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400e9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:34.820 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.820+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:34.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:34.820+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:35.221+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:35.221+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400e9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:35.221+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:36.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:36.022+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:36.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:36.022+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400e9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:36.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:36.022+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:37.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:37.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400e9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:37.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 0x7f359400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:37.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35940069e0 2023-07-02T09:10:37.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594006df0 2023-07-02T09:10:37.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:37.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:37.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.620+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:37.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.821+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:37.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.821+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:37.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:37.821+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:38.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:38.222+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:38.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:38.222+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:38.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:38.222+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:39.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:39.023+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:39.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:39.023+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:39.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:39.023+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:40.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 msgr2=0x7f359400dbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:40.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594006df0 0x7f359400dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:40.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:40.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:40.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594006df0 2023-07-02T09:10:40.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f3594007730 2023-07-02T09:10:40.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:40.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:40.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.620+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:40.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.821+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:40.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.821+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:40.821 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:40.821+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:41.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:41.222+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:41.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:41.222+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:41.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:41.222+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:42.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:42.023+0000 7f35a77fe700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:42.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:42.023+0000 7f35a7fff700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:42.023 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:42.023+0000 7f35a7fff700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.620+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.620+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3594007730 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.620+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594006df0 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.620+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.620+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a809e260 con 0x7f35940069e0 2023-07-02T09:10:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.620+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f359400e9f0 2023-07-02T09:10:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.620+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.620+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.621+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:43.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.822+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:43.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.822+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:43.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:43.822+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:44.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:44.223+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:44.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:44.223+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:44.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:44.223+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:45.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:45.024+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:45.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:45.024+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:45.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:45.024+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:46.621 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a5ffb700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f3594014720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:46.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f3594014720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-02T09:10:46.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a5ffb700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f35940069e0 legacy=0x7f359400cbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-02T09:10:46.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a5ffb700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f35940064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-02T09:10:46.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a5ffb700 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8132a20 con 0x7f3594007730 2023-07-02T09:10:46.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a5ffb700 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80a1640 con 0x7f359400e9f0 2023-07-02T09:10:46.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:46.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007730 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:46.623 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.621+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-02T09:10:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.822+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.822+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007730 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:46.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:46.822+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-02T09:10:47.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:47.223+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:47.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:47.223+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007730 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:47.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:47.223+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-02T09:10:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:48.024+0000 7f35a7fff700 1 -- >> v1:172.21.15.103:6789/0 conn(0x7f3594007730 legacy=0x7f359400d2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2023-07-02T09:10:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:48.024+0000 7f35a77fe700 1 -- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 msgr2=0x7f35940064a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.103:3300/0 2023-07-02T09:10:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2023-07-02T09:10:48.024+0000 7f35a77fe700 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f359400e9f0 0x7f35940064a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-02T09:10:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-07-02T09:10:49.508 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1938, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1496, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-07-02T09:10:49.509 INFO:teuthology.misc:Shutting down mds daemons... 2023-07-02T09:10:49.510 INFO:teuthology.misc:Shutting down osd daemons... 2023-07-02T09:10:49.510 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-07-02T09:10:49.511 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T09:10:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T09:10:49.512 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1938, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1496, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2023-07-02T09:10:49.512 INFO:tasks.ceph.osd.0:Stopped 2023-07-02T09:10:49.513 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-07-02T09:10:49.513 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T09:10:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T09:10:49.514 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1938, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1496, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2023-07-02T09:10:49.515 INFO:tasks.ceph.osd.1:Stopped 2023-07-02T09:10:49.515 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-07-02T09:10:49.516 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T09:10:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T09:10:49.516 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1938, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1496, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2023-07-02T09:10:49.517 INFO:tasks.ceph.osd.2:Stopped 2023-07-02T09:10:49.518 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-07-02T09:10:49.518 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T09:10:49.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T09:10:49.519 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1938, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1496, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log' 2023-07-02T09:10:49.520 INFO:tasks.ceph.osd.3:Stopped 2023-07-02T09:10:49.520 INFO:teuthology.misc:Shutting down mgr daemons... 2023-07-02T09:10:49.520 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-07-02T09:10:49.521 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T09:10:49.521 INFO:tasks.ceph.mgr.x:Stopped 2023-07-02T09:10:49.522 INFO:teuthology.misc:Shutting down mon daemons... 2023-07-02T09:10:49.522 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-07-02T09:10:49.523 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T09:10:49.523 INFO:tasks.ceph.mon.a:Stopped 2023-07-02T09:10:49.524 INFO:tasks.ceph:Checking cluster log for badness... 2023-07-02T09:10:49.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-07-02T09:10:49.561 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi103.front.sepia.ceph.com 2023-07-02T09:10:49.562 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-07-02T09:10:49.700 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi103.front.sepia.ceph.com 2023-07-02T09:10:49.701 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-07-02T09:10:49.772 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi164.front.sepia.ceph.com 2023-07-02T09:10:49.772 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-07-02T09:10:49.949 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi164.front.sepia.ceph.com 2023-07-02T09:10:49.950 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-07-02T09:10:50.023 INFO:tasks.ceph:Archiving mon data... 2023-07-02T09:10:50.024 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293/data/mon.a.tgz 2023-07-02T09:10:50.025 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2023-07-02T09:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.XaggPk0NvF 2023-07-02T09:10:50.046 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.XaggPk0NvF 2023-07-02T09:10:50.161 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.XaggPk0NvF 2023-07-02T09:10:50.249 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.XaggPk0NvF is 206KB 2023-07-02T09:10:50.316 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.XaggPk0NvF 2023-07-02T09:10:50.333 INFO:tasks.ceph:Cleaning ceph cluster... 2023-07-02T09:10:50.334 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-07-02T09:10:50.375 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-07-02T09:10:50.408 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-07-02T09:10:50.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-07-02T09:10:50.442 DEBUG:teuthology.orchestra.run.smithi164:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-07-02T09:10:50.467 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-07-02T09:10:50.480 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-07-02T09:10:50.487 INFO:tasks.ceph:Archiving crash dumps... 2023-07-02T09:10:50.489 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293/remote/smithi103/crash 2023-07-02T09:10:50.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-07-02T09:10:50.540 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293/remote/smithi164/crash 2023-07-02T09:10:50.541 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-07-02T09:10:50.577 INFO:tasks.ceph:Compressing logs... 2023-07-02T09:10:50.577 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-02T09:10:50.584 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-02T09:10:50.608 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116097.log 2023-07-02T09:10:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120425.log 2023-07-02T09:10:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119396.log 2023-07-02T09:10:50.611 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.116097.log.gz 2023-07-02T09:10:50.611 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110854.log 2023-07-02T09:10:50.611 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120425.log: /var/log/ceph/ceph-client.admin.119396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113624.log 2023-07-02T09:10:50.611 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.120425.log.gz 2023-07-02T09:10:50.611 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.119396.log.gz 2023-07-02T09:10:50.612 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111808.log 2023-07-02T09:10:50.612 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.110854.log.gz 2023-07-02T09:10:50.612 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123038.log 2023-07-02T09:10:50.612 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.113624.log.gz 2023-07-02T09:10:50.612 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121021.log 2023-07-02T09:10:50.612 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.123038.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.111808.log.gz 2023-07-02T09:10:50.612 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114466.log 2023-07-02T09:10:50.612 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.123038.log.gz 2023-07-02T09:10:50.613 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117575.log 2023-07-02T09:10:50.613 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.121021.log.gz 2023-07-02T09:10:50.613 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122891.log 2023-07-02T09:10:50.619 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117575.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.114466.log.gz 2023-07-02T09:10:50.619 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118710.log 2023-07-02T09:10:50.619 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.117575.log.gz 2023-07-02T09:10:50.619 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115852.log 2023-07-02T09:10:50.619 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.122891.log.gz 2023-07-02T09:10:50.619 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110485.log 2023-07-02T09:10:50.619 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.118710.log.gz 2023-07-02T09:10:50.620 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116693.log 2023-07-02T09:10:50.620 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.115852.log.gz 2023-07-02T09:10:50.620 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114809.log 2023-07-02T09:10:50.620 INFO:teuthology.orchestra.run.smithi103.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.110485.log.gz 2023-07-02T09:10:50.620 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121217.log 2023-07-02T09:10:50.620 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114809.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.116693.log.gz 2023-07-02T09:10:50.620 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111690.log 2023-07-02T09:10:50.621 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.114809.log.gz 2023-07-02T09:10:50.621 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112641.log 2023-07-02T09:10:50.621 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.121217.log.gz 2023-07-02T09:10:50.621 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122246.log 2023-07-02T09:10:50.621 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112641.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.111690.log.gz 2023-07-02T09:10:50.621 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116146.log 2023-07-02T09:10:50.621 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.112641.log.gz 2023-07-02T09:10:50.621 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110204.log 2023-07-02T09:10:50.622 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.122246.log.gz 2023-07-02T09:10:50.622 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118465.log 2023-07-02T09:10:50.622 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110204.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.116146.log.gz 2023-07-02T09:10:50.622 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116448.log 2023-07-02T09:10:50.622 INFO:teuthology.orchestra.run.smithi103.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.110204.log.gz 2023-07-02T09:10:50.622 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118906.log 2023-07-02T09:10:50.622 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.118465.log.gz 2023-07-02T09:10:50.622 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117330.log 2023-07-02T09:10:50.623 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.116448.log.gz 2023-07-02T09:10:50.623 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114221.log 2023-07-02T09:10:50.623 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.118906.log.gz 2023-07-02T09:10:50.623 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112473.log 2023-07-02T09:10:50.623 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.117330.log.gz 2023-07-02T09:10:50.623 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113772.log 2023-07-02T09:10:50.623 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112473.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.114221.log.gz 2023-07-02T09:10:50.623 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115753.log 2023-07-02T09:10:50.624 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.112473.log.gz 2023-07-02T09:10:50.624 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112840.log 2023-07-02T09:10:50.624 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.113772.log.gz 2023-07-02T09:10:50.624 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112055.log 2023-07-02T09:10:50.624 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112840.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.115753.log.gz 2023-07-02T09:10:50.624 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113232.log 2023-07-02T09:10:50.624 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.112840.log.gz 2023-07-02T09:10:50.625 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112789.log 2023-07-02T09:10:50.625 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.112055.log.gz 2023-07-02T09:10:50.625 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118759.log 2023-07-02T09:10:50.625 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.113232.log.gz 2023-07-02T09:10:50.625 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117771.log 2023-07-02T09:10:50.625 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.112789.log.gz 2023-07-02T09:10:50.625 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110261.log 2023-07-02T09:10:50.625 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.118759.log.gz 2023-07-02T09:10:50.626 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112396.log 2023-07-02T09:10:50.626 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110261.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.117771.log.gz 2023-07-02T09:10:50.626 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112226.log 2023-07-02T09:10:50.626 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.110261.log.gz 2023-07-02T09:10:50.626 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119102.log 2023-07-02T09:10:50.626 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.112396.log.gz 2023-07-02T09:10:50.626 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120082.log 2023-07-02T09:10:50.648 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.112226.log.gz 2023-07-02T09:10:50.648 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119837.log 2023-07-02T09:10:50.649 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.119102.log.gz 2023-07-02T09:10:50.649 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114123.log 2023-07-02T09:10:50.649 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.120082.log.gz 2023-07-02T09:10:50.649 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118359.log 2023-07-02T09:10:50.649 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114123.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.119837.log.gz 2023-07-02T09:10:50.649 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120229.log 2023-07-02T09:10:50.649 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.114123.log.gz 2023-07-02T09:10:50.650 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118261.log 2023-07-02T09:10:50.650 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.118359.log.gz 2023-07-02T09:10:50.650 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113379.log 2023-07-02T09:10:50.650 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118261.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.120229.log.gz 2023-07-02T09:10:50.650 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114613.log 2023-07-02T09:10:50.650 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.118261.log.gz 2023-07-02T09:10:50.650 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110310.log 2023-07-02T09:10:50.651 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.113379.log.gz 2023-07-02T09:10:50.651 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111271.log 2023-07-02T09:10:50.651 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110310.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.114613.log.gz 2023-07-02T09:10:50.651 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120629.log 2023-07-02T09:10:50.651 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.110310.log.gz 2023-07-02T09:10:50.651 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111220.log 2023-07-02T09:10:50.651 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.111271.log.gz 2023-07-02T09:10:50.651 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123136.log 2023-07-02T09:10:50.652 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.120629.log.gz 2023-07-02T09:10:50.652 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110484.log 2023-07-02T09:10:50.652 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.111220.log.gz 2023-07-02T09:10:50.652 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.123136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119298.log 2023-07-02T09:10:50.652 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.123136.log.gz 2023-07-02T09:10:50.652 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114319.log 2023-07-02T09:10:50.652 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119298.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110484.log.gz 2023-07-02T09:10:50.653 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117379.log 2023-07-02T09:10:50.653 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.119298.log.gz 2023-07-02T09:10:50.653 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116987.log 2023-07-02T09:10:50.653 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.114319.log.gz 2023-07-02T09:10:50.653 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119690.log 2023-07-02T09:10:50.653 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116987.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.117379.log.gz 2023-07-02T09:10:50.653 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115263.log 2023-07-02T09:10:50.653 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.116987.log.gz 2023-07-02T09:10:50.654 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123185.log 2023-07-02T09:10:50.654 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.119690.log.gz 2023-07-02T09:10:50.654 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116195.log 2023-07-02T09:10:50.654 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.115263.log.gz 2023-07-02T09:10:50.654 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.123185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120572.log 2023-07-02T09:10:50.654 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.123185.log.gz 2023-07-02T09:10:50.654 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115557.log 2023-07-02T09:10:50.654 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.116195.log.gz 2023-07-02T09:10:50.655 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121854.log 2023-07-02T09:10:50.655 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.120572.log.gz 2023-07-02T09:10:50.655 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111107.log 2023-07-02T09:10:50.655 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.115557.log.gz 2023-07-02T09:10:50.655 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121952.log 2023-07-02T09:10:50.655 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.121854.log.gz 2023-07-02T09:10:50.655 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116244.log 2023-07-02T09:10:50.656 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.111107.log.gz 2023-07-02T09:10:50.656 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114907.log 2023-07-02T09:10:50.656 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116244.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.121952.log.gz 2023-07-02T09:10:50.656 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117281.log 2023-07-02T09:10:50.656 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.116244.log.gz 2023-07-02T09:10:50.656 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114417.log 2023-07-02T09:10:50.656 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.114907.log.gz 2023-07-02T09:10:50.656 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115165.log 2023-07-02T09:10:50.657 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.117281.log.gz 2023-07-02T09:10:50.657 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113428.log 2023-07-02T09:10:50.657 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.114417.log.gz 2023-07-02T09:10:50.657 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122793.log 2023-07-02T09:10:50.657 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.115165.log.gz 2023-07-02T09:10:50.657 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116546.log 2023-07-02T09:10:50.657 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.113428.log.gz 2023-07-02T09:10:50.657 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111050.log 2023-07-02T09:10:50.658 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.122793.log.gz 2023-07-02T09:10:50.658 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115116.log 2023-07-02T09:10:50.658 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.116546.log.gz 2023-07-02T09:10:50.658 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111049.log 2023-07-02T09:10:50.658 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115116.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.111050.log.gz 2023-07-02T09:10:50.658 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118310.log 2023-07-02T09:10:50.658 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.115116.log.gz 2023-07-02T09:10:50.659 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117085.log 2023-07-02T09:10:50.659 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.111049.log.gz 2023-07-02T09:10:50.659 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112474.log 2023-07-02T09:10:50.659 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.118310.log.gz 2023-07-02T09:10:50.659 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114066.log 2023-07-02T09:10:50.659 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.117085.log.gz 2023-07-02T09:10:50.659 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118065.log 2023-07-02T09:10:50.659 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.112474.log.gz 2023-07-02T09:10:50.660 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122197.log 2023-07-02T09:10:50.660 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.114066.log.gz 2023-07-02T09:10:50.660 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116644.log 2023-07-02T09:10:50.660 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.118065.log.gz 2023-07-02T09:10:50.660 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118857.log 2023-07-02T09:10:50.660 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.122197.log.gz 2023-07-02T09:10:50.660 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119004.log 2023-07-02T09:10:50.660 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.116644.log.gz 2023-07-02T09:10:50.661 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115214.log 2023-07-02T09:10:50.661 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.118857.log.gz 2023-07-02T09:10:50.661 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119004.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-07-02T09:10:50.661 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.119004.log.gz 2023-07-02T09:10:50.661 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113821.log 2023-07-02T09:10:50.661 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.115214.log.gz 2023-07-02T09:10:50.661 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114956.log 2023-07-02T09:10:50.662 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112082.log 2023-07-02T09:10:50.662 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.113821.log.gz 2023-07-02T09:10:50.662 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113919.log 2023-07-02T09:10:50.662 INFO:teuthology.orchestra.run.smithi103.stderr: 88.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-07-02T09:10:50.662 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.114956.log.gz 2023-07-02T09:10:50.662 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114017.log 2023-07-02T09:10:50.662 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121168.log 2023-07-02T09:10:50.662 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.112082.log.gz 2023-07-02T09:10:50.663 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.113919.log.gz 2023-07-02T09:10:50.663 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111272.log 2023-07-02T09:10:50.663 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.114017.log.gz 2023-07-02T09:10:50.663 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119053.log 2023-07-02T09:10:50.663 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.121168.log.gz 2023-07-02T09:10:50.663 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111664.log 2023-07-02T09:10:50.663 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.111272.log.gz 2023-07-02T09:10:50.663 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121070.log 2023-07-02T09:10:50.664 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.119053.log.gz 2023-07-02T09:10:50.664 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114760.log 2023-07-02T09:10:50.664 INFO:teuthology.orchestra.run.smithi103.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.111664.log.gz 2023-07-02T09:10:50.664 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123234.log 2023-07-02T09:10:50.664 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.121070.log.gz 2023-07-02T09:10:50.664 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111494.log 2023-07-02T09:10:50.664 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.114760.log.gz 2023-07-02T09:10:50.665 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.123234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118563.log 2023-07-02T09:10:50.665 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118808.log 2023-07-02T09:10:50.665 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.111494.log.gz 2023-07-02T09:10:50.665 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122148.log 2023-07-02T09:10:50.665 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.118563.log.gz 2023-07-02T09:10:50.665 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111612.log 2023-07-02T09:10:50.665 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118808.log: /var/log/ceph/ceph-client.admin.122148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111076.log 2023-07-02T09:10:50.665 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.118808.log.gz 2023-07-02T09:10:50.666 INFO:teuthology.orchestra.run.smithi103.stderr: 95.8% -- replaced with /var/log/ceph/ceph-client.admin.123234.log.gz 2023-07-02T09:10:50.666 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.122148.log.gz 2023-07-02T09:10:50.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116350.log 2023-07-02T09:10:50.666 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.111612.log.gz 2023-07-02T09:10:50.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116048.log 2023-07-02T09:10:50.666 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.111076.log.gz 2023-07-02T09:10:50.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115704.log 2023-07-02T09:10:50.666 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.116350.log.gz 2023-07-02T09:10:50.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121805.log 2023-07-02T09:10:50.667 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.116048.log.gz 2023-07-02T09:10:50.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115803.log 2023-07-02T09:10:50.667 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.115704.log.gz 2023-07-02T09:10:50.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119151.log 2023-07-02T09:10:50.667 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.121805.log.gz 2023-07-02T09:10:50.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121119.log 2023-07-02T09:10:50.667 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.115803.log.gz 2023-07-02T09:10:50.668 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121364.log 2023-07-02T09:10:50.668 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.119151.log.gz 2023-07-02T09:10:50.668 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121119.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-07-02T09:10:50.668 INFO:teuthology.orchestra.run.smithi103.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.121119.log.gz 2023-07-02T09:10:50.668 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113526.log 2023-07-02T09:10:50.668 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.0.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.121364.log.gz 2023-07-02T09:10:50.668 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117428.log 2023-07-02T09:10:50.669 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119641.log 2023-07-02T09:10:50.669 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.113526.log.gz 2023-07-02T09:10:50.669 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117428.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.117428.log.gz 2023-07-02T09:10:50.669 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116399.log 2023-07-02T09:10:50.669 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114172.log 2023-07-02T09:10:50.669 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.119641.log.gz 2023-07-02T09:10:50.669 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112472.log 2023-07-02T09:10:50.669 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.116399.log.gz 2023-07-02T09:10:50.670 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122842.log 2023-07-02T09:10:50.670 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.114172.log.gz 2023-07-02T09:10:50.670 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116595.log 2023-07-02T09:10:50.670 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112472.log: /var/log/ceph/ceph-client.admin.122842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114662.log 2023-07-02T09:10:50.670 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.112472.log.gz 2023-07-02T09:10:50.670 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.122842.log.gz 2023-07-02T09:10:50.670 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122050.log 2023-07-02T09:10:50.670 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.116595.log.gz 2023-07-02T09:10:50.671 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122687.log 2023-07-02T09:10:50.671 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.114662.log.gz 2023-07-02T09:10:50.671 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112004.log 2023-07-02T09:10:50.671 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122050.log: /var/log/ceph/ceph-client.admin.122687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120874.log 2023-07-02T09:10:50.671 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122050.log.gz 2023-07-02T09:10:50.671 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122687.log.gz 2023-07-02T09:10:50.671 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114858.log 2023-07-02T09:10:50.672 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.112004.log.gz 2023-07-02T09:10:50.672 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119494.log 2023-07-02T09:10:50.672 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.120874.log.gz 2023-07-02T09:10:50.672 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116889.log 2023-07-02T09:10:50.672 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114858.log: /var/log/ceph/ceph-client.admin.119494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112987.log 2023-07-02T09:10:50.672 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.114858.log.gz 2023-07-02T09:10:50.672 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.119494.log.gz 2023-07-02T09:10:50.672 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110969.log 2023-07-02T09:10:50.673 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.116889.log.gz 2023-07-02T09:10:50.673 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111860.log 2023-07-02T09:10:50.673 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112987.log: /var/log/ceph/ceph-client.admin.110969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122491.log 2023-07-02T09:10:50.673 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.112987.log.gz 2023-07-02T09:10:50.673 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.110969.log.gz 2023-07-02T09:10:50.673 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122001.log 2023-07-02T09:10:50.673 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.111860.log.gz 2023-07-02T09:10:50.673 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119543.log 2023-07-02T09:10:50.674 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122491.log.gz 2023-07-02T09:10:50.674 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115361.log 2023-07-02T09:10:50.674 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122001.log.gz 2023-07-02T09:10:50.674 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122736.log 2023-07-02T09:10:50.674 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.119543.log.gz 2023-07-02T09:10:50.674 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109019.log 2023-07-02T09:10:50.674 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.115361.log.gz 2023-07-02T09:10:50.675 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110880.log 2023-07-02T09:10:50.675 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122736.log.gz 2023-07-02T09:10:50.675 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.109019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120327.log 2023-07-02T09:10:50.675 INFO:teuthology.orchestra.run.smithi103.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.109019.log.gz 2023-07-02T09:10:50.675 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117673.log 2023-07-02T09:10:50.675 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.110880.log.gz 2023-07-02T09:10:50.675 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110681.log 2023-07-02T09:10:50.675 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.120327.log.gz 2023-07-02T09:10:50.676 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118661.log 2023-07-02T09:10:50.676 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110681.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.117673.log.gz 2023-07-02T09:10:50.676 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117624.log 2023-07-02T09:10:50.676 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.110681.log.gz 2023-07-02T09:10:50.676 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112739.log 2023-07-02T09:10:50.676 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.118661.log.gz 2023-07-02T09:10:50.676 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122442.log 2023-07-02T09:10:50.676 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.117624.log.gz 2023-07-02T09:10:50.677 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122940.log 2023-07-02T09:10:50.677 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.112739.log.gz 2023-07-02T09:10:50.677 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120678.log 2023-07-02T09:10:50.677 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.122442.log.gz 2023-07-02T09:10:50.677 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120278.log 2023-07-02T09:10:50.677 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.122940.log.gz 2023-07-02T09:10:50.677 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113330.log 2023-07-02T09:10:50.680 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108512.log 2023-07-02T09:10:50.680 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.108512.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-07-02T09:10:50.680 INFO:teuthology.orchestra.run.smithi164.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.108512.log.gz 2023-07-02T09:10:50.681 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-07-02T09:10:50.681 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-07-02T09:10:50.681 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108613.log 2023-07-02T09:10:50.681 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108662.log 2023-07-02T09:10:50.681 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.108613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108562.log 2023-07-02T09:10:50.681 INFO:teuthology.orchestra.run.smithi164.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.108613.log.gz 2023-07-02T09:10:50.681 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.108662.log: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.108662.log.gz 2023-07-02T09:10:50.682 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.108562.log: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.108562.log.gz 2023-07-02T09:10:50.682 INFO:teuthology.orchestra.run.smithi164.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-07-02T09:10:50.683 INFO:teuthology.orchestra.run.smithi164.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-07-02T09:10:50.696 INFO:teuthology.orchestra.run.smithi164.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-07-02T09:10:50.696 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-07-02T09:10:50.696 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m0.047s 2023-07-02T09:10:50.696 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m0.084s 2023-07-02T09:10:50.696 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.016s 2023-07-02T09:10:50.707 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.120678.log.gz 2023-07-02T09:10:50.708 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115950.log 2023-07-02T09:10:50.708 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.120278.log.gz 2023-07-02T09:10:50.708 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118408.log 2023-07-02T09:10:50.708 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.113330.log.gz 2023-07-02T09:10:50.708 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120180.log 2023-07-02T09:10:50.708 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.115950.log.gz 2023-07-02T09:10:50.709 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116938.log 2023-07-02T09:10:50.709 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.118408.log.gz 2023-07-02T09:10:50.709 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119249.log 2023-07-02T09:10:50.709 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.120180.log.gz 2023-07-02T09:10:50.709 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118114.log 2023-07-02T09:10:50.709 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.116938.log.gz 2023-07-02T09:10:50.709 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121315.log 2023-07-02T09:10:50.709 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.119249.log.gz 2023-07-02T09:10:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116791.log 2023-07-02T09:10:50.710 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.118114.log.gz 2023-07-02T09:10:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118612.log 2023-07-02T09:10:50.710 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.121315.log.gz 2023-07-02T09:10:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117183.log 2023-07-02T09:10:50.710 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.116791.log.gz 2023-07-02T09:10:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110155.log 2023-07-02T09:10:50.710 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.118612.log.gz 2023-07-02T09:10:50.711 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118514.log 2023-07-02T09:10:50.711 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.117183.log.gz 2023-07-02T09:10:50.711 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111468.log 2023-07-02T09:10:50.711 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.110155.log.gz 2023-07-02T09:10:50.711 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114368.log 2023-07-02T09:10:50.711 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.118514.log.gz 2023-07-02T09:10:50.711 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112592.log 2023-07-02T09:10:50.712 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.111468.log.gz 2023-07-02T09:10:50.712 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110359.log 2023-07-02T09:10:50.712 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.114368.log.gz 2023-07-02T09:10:50.712 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121903.log 2023-07-02T09:10:50.712 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.112592.log.gz 2023-07-02T09:10:50.712 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113477.log 2023-07-02T09:10:50.712 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.110359.log.gz 2023-07-02T09:10:50.712 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113036.log 2023-07-02T09:10:50.713 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.121903.log.gz 2023-07-02T09:10:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121609.log 2023-07-02T09:10:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113036.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.113477.log.gz 2023-07-02T09:10:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115999.log 2023-07-02T09:10:50.713 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.113036.log.gz 2023-07-02T09:10:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120523.log 2023-07-02T09:10:50.713 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.121609.log.gz 2023-07-02T09:10:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112275.log 2023-07-02T09:10:50.714 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.115999.log.gz 2023-07-02T09:10:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116840.log 2023-07-02T09:10:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112275.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.120523.log.gz 2023-07-02T09:10:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121756.log 2023-07-02T09:10:50.714 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.112275.log.gz 2023-07-02T09:10:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113870.log 2023-07-02T09:10:50.714 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.116840.log.gz 2023-07-02T09:10:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121462.log 2023-07-02T09:10:50.715 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.121756.log.gz 2023-07-02T09:10:50.715 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117869.log 2023-07-02T09:10:50.715 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.113870.log.gz 2023-07-02T09:10:50.715 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119347.log 2023-07-02T09:10:50.715 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.121462.log.gz 2023-07-02T09:10:50.715 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113281.log 2023-07-02T09:10:50.715 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.117869.log.gz 2023-07-02T09:10:50.716 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121266.log 2023-07-02T09:10:50.716 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.119347.log.gz 2023-07-02T09:10:50.716 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111416.log 2023-07-02T09:10:50.716 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.113281.log.gz 2023-07-02T09:10:50.716 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109243.log 2023-07-02T09:10:50.716 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.121266.log.gz 2023-07-02T09:10:50.716 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119592.log 2023-07-02T09:10:50.716 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.111416.log.gz 2023-07-02T09:10:50.717 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.109243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110483.log 2023-07-02T09:10:50.717 INFO:teuthology.orchestra.run.smithi103.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.109243.log.gz 2023-07-02T09:10:50.717 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117526.log 2023-07-02T09:10:50.717 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.119592.log.gz 2023-07-02T09:10:50.717 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119445.log 2023-07-02T09:10:50.717 INFO:teuthology.orchestra.run.smithi103.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.110483.log.gz 2023-07-02T09:10:50.717 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110683.log 2023-07-02T09:10:50.718 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.117526.log.gz 2023-07-02T09:10:50.718 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122589.log 2023-07-02T09:10:50.718 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.119445.log.gz 2023-07-02T09:10:50.718 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121658.log 2023-07-02T09:10:50.718 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.110683.log.gz 2023-07-02T09:10:50.718 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114515.log 2023-07-02T09:10:50.718 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.122589.log.gz/var/log/ceph/ceph-client.admin.121658.log: 2023-07-02T09:10:50.718 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121413.log 2023-07-02T09:10:50.719 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.121658.log.gz 2023-07-02T09:10:50.719 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109218.log 2023-07-02T09:10:50.719 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122344.log 2023-07-02T09:10:50.719 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.114515.log.gz 2023-07-02T09:10:50.719 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.121413.log.gz 2023-07-02T09:10:50.719 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.109218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113673.log 2023-07-02T09:10:50.719 INFO:teuthology.orchestra.run.smithi103.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.109218.log.gz 2023-07-02T09:10:50.719 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-07-02T09:10:50.720 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122344.log: /var/log/ceph/ceph-client.admin.113673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113085.log 2023-07-02T09:10:50.720 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122344.log.gz 2023-07-02T09:10:50.720 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.113673.log.gz 2023-07-02T09:10:50.720 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119788.log 2023-07-02T09:10:50.720 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111886.log 2023-07-02T09:10:50.720 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.113085.log.gz 2023-07-02T09:10:50.720 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111663.log 2023-07-02T09:10:50.720 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.119788.log.gz 2023-07-02T09:10:50.721 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112225.log 2023-07-02T09:10:50.721 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111663.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.111886.log.gz 2023-07-02T09:10:50.721 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113968.log 2023-07-02T09:10:50.721 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.111663.log.gz 2023-07-02T09:10:50.721 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120776.log 2023-07-02T09:10:50.721 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120972.log 2023-07-02T09:10:50.721 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.112225.log.gz 2023-07-02T09:10:50.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113968.log: /var/log/ceph/ceph-client.admin.120776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111859.log 2023-07-02T09:10:50.722 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.113968.log.gz 2023-07-02T09:10:50.722 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.120776.log.gz 2023-07-02T09:10:50.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120727.log 2023-07-02T09:10:50.722 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.120972.log.gz 2023-07-02T09:10:50.722 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121707.log 2023-07-02T09:10:50.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111859.log: /var/log/ceph/ceph-client.admin.120727.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.111859.log.gz 2023-07-02T09:10:50.722 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118163.log 2023-07-02T09:10:50.723 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.120727.log.gz 2023-07-02T09:10:50.723 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112889.log 2023-07-02T09:10:50.723 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117232.log 2023-07-02T09:10:50.723 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.121707.log.gz 2023-07-02T09:10:50.723 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.118163.log.gz 2023-07-02T09:10:50.723 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120474.log 2023-07-02T09:10:50.723 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.112889.log.gz 2023-07-02T09:10:50.723 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117820.log 2023-07-02T09:10:50.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120474.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.117232.log.gz 2023-07-02T09:10:50.724 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113722.log 2023-07-02T09:10:50.724 INFO:teuthology.orchestra.run.smithi103.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-07-02T09:10:50.724 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.120474.log.gz 2023-07-02T09:10:50.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122540.log 2023-07-02T09:10:50.724 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.117820.log.gz 2023-07-02T09:10:50.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110684.log 2023-07-02T09:10:50.724 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.113722.log.gz 2023-07-02T09:10:50.725 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111467.log 2023-07-02T09:10:50.725 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.122540.log.gz 2023-07-02T09:10:50.725 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110486.log 2023-07-02T09:10:50.725 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.110684.log.gz 2023-07-02T09:10:50.725 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112690.log 2023-07-02T09:10:50.725 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.111467.log.gz 2023-07-02T09:10:50.725 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120033.log 2023-07-02T09:10:50.725 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112690.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110486.log.gz 2023-07-02T09:10:50.726 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115067.log 2023-07-02T09:10:50.726 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.112690.log.gz 2023-07-02T09:10:50.726 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117477.log 2023-07-02T09:10:50.726 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.120033.log.gz 2023-07-02T09:10:50.726 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114711.log 2023-07-02T09:10:50.726 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.115067.log.gz 2023-07-02T09:10:50.726 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116742.log 2023-07-02T09:10:50.727 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.117477.log.gz 2023-07-02T09:10:50.727 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110682.log 2023-07-02T09:10:50.727 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.114711.log.gz 2023-07-02T09:10:50.727 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116742.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-07-02T09:10:50.727 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.116742.log.gz 2023-07-02T09:10:50.727 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121560.log 2023-07-02T09:10:50.727 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.110682.log.gz 2023-07-02T09:10:50.727 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117722.log 2023-07-02T09:10:50.728 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.121560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113134.log 2023-07-02T09:10:50.728 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.121560.log.gz 2023-07-02T09:10:50.728 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117722.log: 89.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-07-02T09:10:50.728 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115606.log 2023-07-02T09:10:50.728 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.117722.log.gz 2023-07-02T09:10:50.728 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113183.log 2023-07-02T09:10:50.728 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113134.log: /var/log/ceph/ceph-client.admin.115606.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.113134.log.gz 2023-07-02T09:10:50.728 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123288.log 2023-07-02T09:10:50.729 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.115606.log.gz 2023-07-02T09:10:50.729 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119739.log 2023-07-02T09:10:50.729 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.113183.log.gz 2023-07-02T09:10:50.729 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.123288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111299.log 2023-07-02T09:10:50.729 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120923.log 2023-07-02T09:10:50.729 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.119739.log.gz 2023-07-02T09:10:50.729 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.111299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117967.log 2023-07-02T09:10:50.729 INFO:teuthology.orchestra.run.smithi103.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.111299.log.gz 2023-07-02T09:10:50.730 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118212.log 2023-07-02T09:10:50.730 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120923.log: /var/log/ceph/ceph-client.admin.117967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120131.log 2023-07-02T09:10:50.730 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.120923.log.gz 2023-07-02T09:10:50.730 INFO:teuthology.orchestra.run.smithi103.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.admin.123288.log.gz 2023-07-02T09:10:50.730 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.117967.log.gz 2023-07-02T09:10:50.730 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119886.log 2023-07-02T09:10:50.730 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.118212.log.gz 2023-07-02T09:10:50.730 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110086.log 2023-07-02T09:10:50.731 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120131.log: /var/log/ceph/ceph-client.admin.119886.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.120131.log.gz 2023-07-02T09:10:50.731 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112278.log 2023-07-02T09:10:50.731 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.119886.log.gz 2023-07-02T09:10:50.731 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119935.log 2023-07-02T09:10:50.731 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.110086.log.gz 2023-07-02T09:10:50.731 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116301.log 2023-07-02T09:10:50.731 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.112278.log.gz 2023-07-02T09:10:50.732 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115508.log 2023-07-02T09:10:50.732 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.119935.log.gz 2023-07-02T09:10:50.732 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122393.log 2023-07-02T09:10:50.732 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.116301.log.gz 2023-07-02T09:10:50.732 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119984.log 2023-07-02T09:10:50.732 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.115508.log.gz 2023-07-02T09:10:50.732 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121511.log 2023-07-02T09:10:50.732 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122393.log.gz 2023-07-02T09:10:50.733 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118955.log 2023-07-02T09:10:50.733 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.119984.log.gz 2023-07-02T09:10:50.733 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.121511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115901.log 2023-07-02T09:10:50.733 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.121511.log.gz 2023-07-02T09:10:50.733 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118955.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.106390.log 2023-07-02T09:10:50.733 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.118955.log.gz 2023-07-02T09:10:50.733 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116497.log 2023-07-02T09:10:50.733 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.115901.log.gz 2023-07-02T09:10:50.734 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.tmp-client.admin.106390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120825.log 2023-07-02T09:10:50.734 INFO:teuthology.orchestra.run.smithi103.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.106390.log.gz 2023-07-02T09:10:50.734 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.116497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117918.log 2023-07-02T09:10:50.734 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.116497.log.gz 2023-07-02T09:10:50.734 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.120825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112938.log 2023-07-02T09:10:50.734 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.120825.log.gz 2023-07-02T09:10:50.734 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110828.log 2023-07-02T09:10:50.734 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112938.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.117918.log.gz 2023-07-02T09:10:50.735 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117036.log 2023-07-02T09:10:50.735 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.112938.log.gz 2023-07-02T09:10:50.735 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.110828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123087.log 2023-07-02T09:10:50.735 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.110828.log.gz 2023-07-02T09:10:50.735 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123341.log 2023-07-02T09:10:50.735 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.123087.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.117036.log.gz 2023-07-02T09:10:50.735 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-07-02T09:10:50.735 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.123087.log.gz 2023-07-02T09:10:50.736 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.123341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117134.log 2023-07-02T09:10:50.736 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119200.log 2023-07-02T09:10:50.736 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.117134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114564.log 2023-07-02T09:10:50.736 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.117134.log.gz 2023-07-02T09:10:50.736 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.119200.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.119200.log.gz 2023-07-02T09:10:50.736 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.114564.log: 95.9% -- replaced with /var/log/ceph/ceph-client.admin.123341.log.gz 2023-07-02T09:10:50.736 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.114564.log.gz 2023-07-02T09:10:50.737 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108933.log 2023-07-02T09:10:50.737 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112056.log 2023-07-02T09:10:50.737 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.108933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113575.log 2023-07-02T09:10:50.737 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.112056.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.108933.log.gz 2023-07-02T09:10:50.737 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122989.log 2023-07-02T09:10:50.737 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.112056.log.gz 2023-07-02T09:10:50.737 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.113575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114270.log 2023-07-02T09:10:50.737 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.113575.log.gz 2023-07-02T09:10:50.738 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115459.log 2023-07-02T09:10:50.738 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122989.log: /var/log/ceph/ceph-client.admin.114270.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122989.log.gz 2023-07-02T09:10:50.738 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115018.log 2023-07-02T09:10:50.738 INFO:teuthology.orchestra.run.smithi103.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.114270.log.gz 2023-07-02T09:10:50.738 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122099.log 2023-07-02T09:10:50.738 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118016.log 2023-07-02T09:10:50.738 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.115459.log.gz 2023-07-02T09:10:50.738 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122099.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.115018.log.gz 2023-07-02T09:10:50.739 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122638.log 2023-07-02T09:10:50.739 INFO:teuthology.orchestra.run.smithi103.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.122099.log.gz 2023-07-02T09:10:50.739 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.118016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115655.log 2023-07-02T09:10:50.739 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.118016.log.gz 2023-07-02T09:10:50.739 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115312.log 2023-07-02T09:10:50.739 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122638.log: /var/log/ceph/ceph-client.admin.115655.log: gzip -5 86.0%--verbose -- /var/log/ceph/ceph-client.admin.115410.log -- replaced with /var/log/ceph/ceph-client.admin.122638.log.gz 2023-07-02T09:10:50.739 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-07-02T09:10:50.739 INFO:teuthology.orchestra.run.smithi103.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.115655.log.gz 2023-07-02T09:10:50.740 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120376.log 2023-07-02T09:10:50.740 INFO:teuthology.orchestra.run.smithi103.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.115312.log.gz 2023-07-02T09:10:50.740 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122295.log 2023-07-02T09:10:50.740 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.115410.log: /var/log/ceph/ceph-client.admin.120376.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.115410.log.gz 2023-07-02T09:10:50.740 INFO:teuthology.orchestra.run.smithi103.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.120376.log.gz 2023-07-02T09:10:50.740 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.122295.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.122295.log.gz 2023-07-02T09:10:50.740 INFO:teuthology.orchestra.run.smithi103.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-07-02T09:10:50.822 INFO:teuthology.orchestra.run.smithi103.stderr: 89.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-07-02T09:10:50.823 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-07-02T09:10:50.823 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m0.225s 2023-07-02T09:10:50.824 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m0.301s 2023-07-02T09:10:50.824 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.139s 2023-07-02T09:10:50.824 INFO:tasks.ceph:Archiving logs... 2023-07-02T09:10:50.825 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293/remote/smithi103/log 2023-07-02T09:10:50.826 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-02T09:10:51.098 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293/remote/smithi164/log 2023-07-02T09:10:51.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-02T09:10:51.168 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1938, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1496, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-07-02T09:10:52.295 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=09f8cee8f80747bdb6457f698db0b1aa Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1938, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1496, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_00edceddba022d4d35c4fe2bf8ae4808b752654c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-07-02T09:10:52.320 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-07-02T09:10:52.330 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-02T09:10:52.359 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-02T09:10:52.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-02T09:10:52.385 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-02T09:10:52.475 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-02T09:10:52.485 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-02T09:10:52.486 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T09:10:52.488 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T09:10:52.506 INFO:teuthology.orchestra.run.smithi103.stderr:bash: ntpq: command not found 2023-07-02T09:10:52.507 INFO:teuthology.orchestra.run.smithi164.stderr:bash: ntpq: command not found 2023-07-02T09:10:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T09:10:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2023-07-02T09:10:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:^- hv01.front.sepia.ceph.com 2 6 377 20 +1495us[+1495us] +/- 268ms 2023-07-02T09:10:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 +53us[ +86us] +/- 53ms 2023-07-02T09:10:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 -61us[ -29us] +/- 78ms 2023-07-02T09:10:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T09:10:52.514 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T09:10:52.514 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2023-07-02T09:10:52.514 INFO:teuthology.orchestra.run.smithi164.stdout:^- hv01.front.sepia.ceph.com 2 6 377 21 +1502us[+1506us] +/- 268ms 2023-07-02T09:10:52.515 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 +61us[ +65us] +/- 53ms 2023-07-02T09:10:52.515 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -72us[ -72us] +/- 78ms 2023-07-02T09:10:52.515 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T09:10:52.515 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-02T09:10:52.524 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-02T09:10:52.525 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-02T09:10:52.568 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-02T09:10:52.668 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-02T09:10:52.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T09:10:52.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T09:10:52.796 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T09:10:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T09:10:52.829 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-02T09:10:52.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-02T09:10:52.884 INFO:teuthology.task.internal:Duration was 1843.224365 seconds 2023-07-02T09:10:52.884 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-02T09:10:52.934 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-02T09:10:52.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-02T09:10:52.937 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-02T09:10:52.982 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T09:10:52.985 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T09:10:53.359 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-02T09:10:53.359 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2023-07-02T09:10:53.360 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-02T09:10:53.383 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2023-07-02T09:10:53.383 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-02T09:10:53.405 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-02T09:10:53.406 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-02T09:10:53.425 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-02T09:10:53.482 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-02T09:10:53.483 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-02T09:10:53.505 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-02T09:10:53.586 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-02T09:10:53.596 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-02T09:10:53.597 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-02T09:10:53.624 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-02T09:10:53.651 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-02T09:10:53.661 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-02T09:10:53.666 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-02T09:10:53.690 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2023-07-02T09:10:53.717 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2023-07-02T09:10:53.740 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-02T09:10:53.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T09:10:53.776 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-02T09:10:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T09:10:53.797 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-02T09:10:53.808 INFO:teuthology.task.internal:Transferring archived files... 2023-07-02T09:10:53.809 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293/remote/smithi103 2023-07-02T09:10:53.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-02T09:10:53.852 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293/remote/smithi164 2023-07-02T09:10:53.852 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-02T09:10:53.887 INFO:teuthology.task.internal:Removing archive directory... 2023-07-02T09:10:53.887 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-02T09:10:53.889 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-02T09:10:53.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-02T09:10:53.951 INFO:teuthology.task.internal:Not uploading archives. 2023-07-02T09:10:53.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-02T09:10:53.977 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-02T09:10:53.978 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-02T09:10:53.980 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-02T09:10:53.995 INFO:teuthology.orchestra.run.smithi103.stdout: 525343 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 2 09:10 /home/ubuntu/cephtest 2023-07-02T09:10:53.997 INFO:teuthology.orchestra.run.smithi164.stdout: 525343 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 2 09:10 /home/ubuntu/cephtest 2023-07-02T09:10:53.998 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-02T09:10:54.027 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-02T09:10:54.127 INFO:teuthology.nuke:Checking targets against current locks 2023-07-02T09:10:54.157 DEBUG:teuthology.nuke:shortname: smithi103 2023-07-02T09:10:54.158 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-02T09:10:54.184 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 08:25:11.161871', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOqS6e1wxD0IKfQGrjPbQM7M7V4/e4hjGoWI/J5FO15c51f7uUO96HBA/cGQVUVNUmlre9YDyklL2gyLumPXPMBNP7LF4W9ordzlTcNmfKk7eMOQzbpGdVrDmPmb4jSA72Bm2LTYvspP0U1c7eFXvjM4mRYjHjl5IsdCTbdTSFhtR+pYPw6pk/zOX6IuISt9HALOtvguv3x/5x3q1+C7mUFIZMeJUjEoBrY2VHmoPtiWdeqs6PplgEwPBKUqG0PaYhnvMMN2yFrieUj4oAfEiUQ/VXMB1sS9cBCWCA2DgX4FhcF4NMuE96EPPjorl6hx6J43GWGEQHKyMTP/ptmuuGVDOInrAvlgJE5BpNKrtarM1TDFfCJy/MqwTVT/1L8n3wBIug6G1yis0lgmYIoZrrR4xmkThpiqB1WGkidBgeh/Ye2Gwv8JjELSuk+c4UH1PUW49yVeqQ4bCj/aKj6l+3V/Ty9vQa5QivjsgEqUs1SWtMfOcgBGpyH6nxmutQVoE='} 2023-07-02T09:10:54.187 DEBUG:teuthology.nuke:shortname: smithi164 2023-07-02T09:10:54.187 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-02T09:10:54.207 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-07-02_08:25:06-crimson-rados-wip-yingxin-msgr-multi-core-crimson-10-distro-crimson-smithi/7324293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 08:25:11.160107', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj/UFFLKldYHPXfTfKh85+du+x3iJh4dlxPOVHYlmglEVLCsVpbRknPbArvu1MyXmisW5Lknb9KYL7Xi6h+EvZfYiLpDdD4CoJAkrb73ldcQ9OnH5tPac+mDM/D4AxS5zv2uiRXWiBqgCS2vwiUpdh0eZFnCtysnBpzAYU5tyjdjKdGWjCvD2i+5DfJs/ciqeDW3coTuRx2GjwSYEwGlQ3w3toVgzayP25ZnzYIrK9sOQD4ZbxBnoWbt9//GEEcQ9G7DEP3g7tfuQAjy655jwS0R6eedNDDTFqrBVqVvKX2hZvGl8icnnSmeUzi7srcIkH7Xmrl98Jn4CR39/iwlcJsEk20+IanaAMwwmnV6leKhaDOExYc2bsk9/ZTHuv5Ufi8gRQbkwLTZoVIrveYsgXCCwf4lScUiKnTqJ9QhVzf9ZbOMXL/HYChzi1PmaYKX7d4bRqkyICyEGChqYqTWyB/DIURFVgkkP69UCBo+YoyyoKpn710e+1I/YG5VOe9xE='} 2023-07-02T09:10:54.240 INFO:teuthology.orchestra.console:Power off smithi103 2023-07-02T09:10:54.241 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-02T09:10:54.265 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T09:10:54.284 INFO:teuthology.orchestra.console:Power off smithi164 2023-07-02T09:10:54.285 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-02T09:10:54.307 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T09:10:58.297 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T09:10:58.328 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T09:11:02.412 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T09:11:02.442 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T09:11:02.626 INFO:teuthology.orchestra.console:Power off for smithi103 completed 2023-07-02T09:11:06.561 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T09:11:06.772 INFO:teuthology.orchestra.console:Power off for smithi164 completed 2023-07-02T09:11:06.877 INFO:teuthology.run:Summary data: description: crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} duration: 1843.2243649959564 failure_reason: 'Command failed on smithi103 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json''' flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=09f8cee8f80747bdb6457f698db0b1aa status: fail success: false 2023-07-02T09:11:06.878 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-02T09:11:06.968 INFO:teuthology.run:FAIL