2024-08-16T02:25:18.533 INFO:root:teuthology version: 1.2.0 2024-08-16T02:25:18.549 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-16T02:25:18.668 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420 branch: wip-yingxin-testing-2024-08-15-1551 description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7857420' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yingxin-testing-2024-08-15-1551 ceph: conf: client: debug ms: 1 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 workunit: branch: wip-yingxin-testing-2024-08-15-1551 env: CRIMSON_COMPAT: '1' sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-yingxin-testing-2024-08-15-1551 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 targets: smithi133.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBkKGkigL0azSs8Ojv+rogQ7hOn1HRKC6g2yWXMDId1EIKYb03B3s0YUmeL9qbrrcUEgEqgYc3hRLDJRjA2HIjs= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsJzJKq6v8ELjQkdDZGhIL3RYLFtefFQhp2GXslHlmUNW3mbUqOMgrqHE/i167yzNoINffqqQIuALeW9I/lqOs= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rados/test.sh --crimson - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-16_01:59:29 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-16T02:25:18.668 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa; will attempt to use it 2024-08-16T02:25:18.669 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks 2024-08-16T02:25:18.669 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-16T02:25:18.671 INFO:teuthology.task.internal:Checking packages... 2024-08-16T02:25:18.693 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '6f386a3f0b1fe6a4d7219b54bf963d790bc84004' 2024-08-16T02:25:18.694 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-16T02:25:18.694 INFO:teuthology.packaging:ref: None 2024-08-16T02:25:18.694 INFO:teuthology.packaging:tag: None 2024-08-16T02:25:18.694 INFO:teuthology.packaging:branch: wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:25:18.694 INFO:teuthology.packaging:sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:25:18.694 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:25:18.833 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5812.g6f386a3f 2024-08-16T02:25:18.834 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-16T02:25:18.841 INFO:teuthology.task.internal:no buildpackages task found 2024-08-16T02:25:18.841 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-16T02:25:18.848 INFO:teuthology.task.internal:Saving configuration 2024-08-16T02:25:18.860 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-16T02:25:18.871 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-16T02:25:18.897 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-16 02:20:18.861625', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBkKGkigL0azSs8Ojv+rogQ7hOn1HRKC6g2yWXMDId1EIKYb03B3s0YUmeL9qbrrcUEgEqgYc3hRLDJRjA2HIjs='} 2024-08-16T02:25:18.922 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-16 02:20:18.860416', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsJzJKq6v8ELjQkdDZGhIL3RYLFtefFQhp2GXslHlmUNW3mbUqOMgrqHE/i167yzNoINffqqQIuALeW9I/lqOs='} 2024-08-16T02:25:18.922 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-16T02:25:18.929 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-08-16T02:25:18.929 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-08-16T02:25:18.930 INFO:teuthology.run_tasks:Running task console_log... 2024-08-16T02:25:19.006 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f20658cbac0>, signals=[15]) 2024-08-16T02:25:19.006 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-16T02:25:19.011 INFO:teuthology.task.internal:Opening connections... 2024-08-16T02:25:19.011 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2024-08-16T02:25:19.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:25:19.088 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-08-16T02:25:19.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:25:19.159 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-16T02:25:19.166 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2024-08-16T02:25:19.184 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2024-08-16T02:25:19.185 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2024-08-16T02:25:19.242 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream" 2024-08-16T02:25:19.242 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="9" 2024-08-16T02:25:19.242 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos" 2024-08-16T02:25:19.242 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora" 2024-08-16T02:25:19.242 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="9" 2024-08-16T02:25:19.242 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el9" 2024-08-16T02:25:19.243 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-16T02:25:19.243 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2024-08-16T02:25:19.243 INFO:teuthology.orchestra.run.smithi133.stdout:LOGO="fedora-logo-icon" 2024-08-16T02:25:19.243 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-16T02:25:19.243 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/" 2024-08-16T02:25:19.243 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-16T02:25:19.243 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-16T02:25:19.243 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-16T02:25:19.244 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2024-08-16T02:25:19.270 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-08-16T02:25:19.288 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-08-16T02:25:19.288 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-08-16T02:25:19.347 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-08-16T02:25:19.347 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-08-16T02:25:19.347 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-08-16T02:25:19.347 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-08-16T02:25:19.347 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-08-16T02:25:19.347 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-08-16T02:25:19.347 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-16T02:25:19.348 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-08-16T02:25:19.348 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-08-16T02:25:19.348 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-16T02:25:19.348 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-08-16T02:25:19.348 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-16T02:25:19.348 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-16T02:25:19.348 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-16T02:25:19.349 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-08-16T02:25:19.370 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-16T02:25:19.379 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-16T02:25:19.403 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-16T02:25:19.403 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2024-08-16T02:25:19.406 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-08-16T02:25:19.423 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-16T02:25:19.429 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-16T02:25:19.429 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2024-08-16T02:25:19.463 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-08-16T02:25:19.495 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-16T02:25:19.555 INFO:teuthology.run_tasks:Running task kernel... 2024-08-16T02:25:19.570 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-08-16T02:25:19.570 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-08-16T02:25:19.570 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-16T02:25:19.571 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-16T02:25:19.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:25:19.587 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2024-08-16T02:25:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:25:19.588 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-08-16T02:25:19.645 INFO:teuthology.orchestra.run.smithi133.stdout:5.14.0-437.el9.x86_64 2024-08-16T02:25:19.645 INFO:teuthology.task.kernel:Running kernel on smithi133: 5.14.0-437.el9.x86_64 2024-08-16T02:25:19.645 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum install -y kernel 2024-08-16T02:25:19.646 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-437.el9.x86_64 2024-08-16T02:25:19.646 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-437.el9.x86_64 2024-08-16T02:25:19.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-08-16T02:25:22.761 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.2 MB 00:01 2024-08-16T02:25:23.994 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.2 MB 00:02 2024-08-16T02:25:25.686 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-08-16T02:25:25.720 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 33 MB/s | 20 MB 00:00 2024-08-16T02:25:29.971 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.6 MB 00:00 2024-08-16T02:25:31.579 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 18 kB 00:00 2024-08-16T02:25:32.145 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.6 MB 00:02 2024-08-16T02:25:32.725 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-08-16T02:25:34.754 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 18 kB 00:01 2024-08-16T02:25:35.941 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-08-16T02:25:38.056 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-16T02:25:40.028 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-16T02:25:40.029 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-16T02:25:40.121 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-16T02:25:40.138 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:25:40.139 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:25:40.139 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:25:40.139 INFO:teuthology.orchestra.run.smithi133.stdout:Install 4 Packages 2024-08-16T02:25:40.139 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:25:40.139 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 84 M 2024-08-16T02:25:40.139 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 126 M 2024-08-16T02:25:40.140 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:25:41.305 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-08-16T02:25:41.490 INFO:teuthology.orchestra.run.smithi133.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.2 MB/s | 1.5 MB 00:01 2024-08-16T02:25:43.253 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-16T02:25:43.253 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-16T02:25:43.345 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:25:43.353 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:25:43.353 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-08-16T02:25:43.353 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-08-16T02:25:43.354 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:25:43.355 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 84 M 2024-08-16T02:25:43.355 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 126 M 2024-08-16T02:25:43.355 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:25:43.814 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 4.5 MB/s | 1.5 MB 00:00 2024-08-16T02:25:45.798 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-08-16T02:25:47.273 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-core-5.14.0-480.el9.x86_64.rpm 4.5 MB/s | 17 MB 00:03 2024-08-16T02:25:48.247 INFO:teuthology.orchestra.run.smithi133.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 2.1 MB/s | 17 MB 00:08 2024-08-16T02:25:49.123 INFO:teuthology.orchestra.run.smithi133.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 4.0 MB/s | 36 MB 00:08 2024-08-16T02:25:57.308 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 2.2 MB/s | 30 MB 00:13 2024-08-16T02:25:57.308 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:25:57.308 INFO:teuthology.orchestra.run.smithi137.stdout:Total 6.1 MB/s | 84 MB 00:13 2024-08-16T02:25:58.044 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:25:58.378 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:25:58.378 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:25:59.361 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:25:59.362 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:26:00.421 INFO:teuthology.orchestra.run.smithi133.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 1.6 MB/s | 30 MB 00:18 2024-08-16T02:26:00.421 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:26:00.422 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.2 MB/s | 84 MB 00:20 2024-08-16T02:26:01.154 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:26:01.505 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:26:01.505 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:26:01.802 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:26:02.498 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:26:02.499 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:26:02.745 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:26:02.847 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:26:05.170 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:26:05.498 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:26:05.634 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:05.904 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:26:06.005 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:26:08.687 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:26:08.839 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:10.423 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:13.696 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:13.831 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:15.413 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:18.714 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:45.020 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:45.060 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:46.189 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:46.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:26:46.190 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:26:46.190 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:46.190 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-16T02:26:46.190 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-16T02:26:46.190 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-16T02:26:46.190 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:26:46.190 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:26:46.575 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:26:46.671 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:46.671 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:26:46.672 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-16T02:26:46.672 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-16T02:26:46.672 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-16T02:26:46.672 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-08-16T02:26:46.672 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-08-16T02:26:46.672 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:26:46.730 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:46.730 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:26:46.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-08-16T02:26:46.771 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-08-16T02:26:46.771 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-08-16T02:26:46.771 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 68.el9 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8264254 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-16T02:26:46.772 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-08-16T02:26:46.773 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-08-16T02:26:46.773 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:26:46.773 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-16T02:26:46.773 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-16T02:26:46.773 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-16T02:26:46.773 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-08-16T02:26:46.773 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:26:46.773 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-08-16T02:26:46.774 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-16T02:26:46.774 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-480.el9.x86_64 2024-08-16T02:26:46.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-16T02:26:47.427 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-08-16T02:26:48.531 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:48.571 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:48.873 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:48.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:26:48.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:26:48.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:48.928 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-16T02:26:48.956 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-08-16T02:26:48.959 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-08-16T02:26:48.975 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.US7LkB5ehe 2024-08-16T02:26:48.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.US7LkB5ehe 2024-08-16T02:26:49.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.US7LkB5ehe 2024-08-16T02:26:49.180 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.US7LkB5ehe is 6KB 2024-08-16T02:26:49.228 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.US7LkB5ehe 2024-08-16T02:26:49.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-08-16T02:26:49.304 INFO:teuthology.orchestra.run.smithi137.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-16T02:26:49.304 INFO:teuthology.orchestra.run.smithi137.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-16T02:26:49.304 INFO:teuthology.orchestra.run.smithi137.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-16T02:26:49.304 INFO:teuthology.orchestra.run.smithi137.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-16T02:26:49.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-16T02:26:49.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-08-16T02:26:49.698 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:49.698 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:26:49.698 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:26:49.698 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:49.698 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-16T02:26:49.698 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-16T02:26:49.698 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-16T02:26:49.699 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:26:49.699 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:26:50.081 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:26:50.171 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:50.172 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:26:50.172 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-16T02:26:50.172 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-16T02:26:50.172 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-16T02:26:50.172 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-16T02:26:50.172 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2024-08-16T02:26:50.173 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:26:50.231 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:50.232 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:26:50.232 DEBUG:teuthology.orchestra.run.smithi133:> sudo rpm -qi grub2-tools 2024-08-16T02:26:50.269 INFO:teuthology.orchestra.run.smithi133.stdout:Name : grub2-tools 2024-08-16T02:26:50.269 INFO:teuthology.orchestra.run.smithi133.stdout:Epoch : 1 2024-08-16T02:26:50.269 INFO:teuthology.orchestra.run.smithi133.stdout:Version : 2.06 2024-08-16T02:26:50.269 INFO:teuthology.orchestra.run.smithi133.stdout:Release : 68.el9 2024-08-16T02:26:50.269 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: x86_64 2024-08-16T02:26:50.269 INFO:teuthology.orchestra.run.smithi133.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Group : Unspecified 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Size : 8264254 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:License : GPLv3+ 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Packager : builder@centos.org 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Vendor : CentOS 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Summary : Support tools for GRUB. 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout:Description : 2024-08-16T02:26:50.270 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:26:50.271 INFO:teuthology.orchestra.run.smithi133.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-16T02:26:50.271 INFO:teuthology.orchestra.run.smithi133.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-16T02:26:50.271 INFO:teuthology.orchestra.run.smithi133.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-16T02:26:50.271 INFO:teuthology.orchestra.run.smithi133.stdout:hardware devices. 2024-08-16T02:26:50.271 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:26:50.271 INFO:teuthology.orchestra.run.smithi133.stdout:This subpackage provides tools for support of all platforms. 2024-08-16T02:26:50.272 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-16T02:26:50.272 INFO:teuthology.task.kernel:Updating grub on smithi133 to boot 5.14.0-480.el9.x86_64 2024-08-16T02:26:50.272 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-16T02:26:50.906 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2024-08-16T02:26:52.219 INFO:teuthology.orchestra.run.smithi133.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-16T02:26:52.259 INFO:teuthology.orchestra.run.smithi133.stderr:done 2024-08-16T02:26:52.262 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2024-08-16T02:26:52.277 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.uNxULxiM3D 2024-08-16T02:26:52.277 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uNxULxiM3D 2024-08-16T02:26:52.342 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.uNxULxiM3D 2024-08-16T02:26:52.472 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.uNxULxiM3D is 6KB 2024-08-16T02:26:52.521 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.uNxULxiM3D 2024-08-16T02:26:52.535 DEBUG:teuthology.orchestra.run.smithi133:> sudo /bin/ls /boot/loader/entries || true 2024-08-16T02:26:52.602 INFO:teuthology.orchestra.run.smithi133.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-16T02:26:52.602 INFO:teuthology.orchestra.run.smithi133.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-16T02:26:52.602 INFO:teuthology.orchestra.run.smithi133.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-16T02:26:52.602 INFO:teuthology.orchestra.run.smithi133.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-16T02:26:52.603 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-16T02:26:52.744 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2024-08-16T02:27:19.470 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-08-16T02:27:19.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-08-16T02:27:19.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:22.750 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-16T02:27:22.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-08-16T02:27:22.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:37.823 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-08-16T02:27:41.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-08-16T02:27:46.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-08-16T02:27:46.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:49.887 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-08-16T02:27:50.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-08-16T02:27:50.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:53.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-08-16T02:28:01.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-08-16T02:28:01.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:28:05.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-08-16T02:28:05.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:28:23.647 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-08-16T02:28:33.843 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-08-16T02:28:34.215 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-08-16T02:28:34.215 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-480.el9.x86_64"... 2024-08-16T02:28:34.215 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-08-16T02:28:34.270 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-480.el9.x86_64 2024-08-16T02:28:34.270 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-16T02:28:34.270 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-16T02:28:34.270 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-16T02:28:35.270 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-08-16T02:28:35.271 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-16T02:28:35.348 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-08-16T02:28:35.389 DEBUG:teuthology.parallel:result is None 2024-08-16T02:28:38.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-08-16T02:28:38.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:28:39.038 DEBUG:teuthology.orchestra.run.smithi133:> true 2024-08-16T02:28:39.391 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-08-16T02:28:39.391 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-16T02:28:39.391 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2024-08-16T02:28:39.406 INFO:teuthology.orchestra.run.smithi133.stdout:5.14.0-480.el9.x86_64 2024-08-16T02:28:39.406 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-16T02:28:39.406 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-16T02:28:39.406 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-16T02:28:40.407 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-16T02:28:40.410 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-16T02:28:40.482 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2024-08-16T02:28:40.517 DEBUG:teuthology.parallel:result is None 2024-08-16T02:28:40.517 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-16T02:28:40.524 INFO:teuthology.task.internal:Creating test directory... 2024-08-16T02:28:40.524 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-16T02:28:40.526 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-16T02:28:40.544 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-16T02:28:40.550 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-16T02:28:40.558 INFO:teuthology.task.internal:Creating archive directory... 2024-08-16T02:28:40.558 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-16T02:28:40.579 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-16T02:28:40.628 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-16T02:28:40.634 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-16T02:28:40.634 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-16T02:28:40.636 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-16T02:28:40.669 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:28:40.679 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:28:40.694 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:28:40.705 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:28:40.707 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-16T02:28:40.713 INFO:teuthology.task.internal:Configuring sudo... 2024-08-16T02:28:40.714 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-16T02:28:40.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-16T02:28:40.771 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-16T02:28:40.780 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-16T02:28:40.781 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:28:40.783 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:28:40.827 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:28:40.869 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:28:40.926 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:28:41.002 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:28:41.042 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:28:41.042 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-16T02:28:41.108 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:28:41.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:28:41.198 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:28:41.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:28:41.320 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:28:41.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-16T02:28:41.385 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2024-08-16T02:28:41.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-08-16T02:28:41.437 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:28:41.484 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:28:41.787 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-16T02:28:41.794 INFO:teuthology.task.internal:Starting timer... 2024-08-16T02:28:41.794 INFO:teuthology.run_tasks:Running task pcp... 2024-08-16T02:28:41.805 INFO:teuthology.run_tasks:Running task selinux... 2024-08-16T02:28:41.815 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2024-08-16T02:28:41.954 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: 2024-08-16T02:28:41.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-08-16T02:28:42.000 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-08-16T02:28:42.002 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-16T02:28:42.002 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2024-08-16T02:28:42.031 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2024-08-16T02:28:42.031 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-08-16T02:28:42.067 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-08-16T02:28:42.067 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi133.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-08-16T02:28:42.067 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:28:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:28:42.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:28:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:28:42.134 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-16T02:28:42.134 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2024-08-16T02:28:42.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-08-16T02:28:42.519 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-16T02:28:42.529 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-16T02:28:42.540 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-16T02:28:42.541 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi133.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-16T02:33:13.918 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi133.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-08-16T02:33:13.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-08-16T02:33:13.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:33:14.000 DEBUG:teuthology.orchestra.run.smithi133:> true 2024-08-16T02:33:14.082 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-08-16T02:33:14.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-08-16T02:33:14.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:33:14.156 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-08-16T02:33:14.230 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-08-16T02:33:14.231 INFO:teuthology.run_tasks:Running task clock... 2024-08-16T02:33:14.243 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-16T02:33:14.244 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-16T02:33:14.244 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:33:14.247 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-16T02:33:14.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:33:14.273 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-16T02:33:14.288 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-16T02:33:14.300 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-16T02:33:14.312 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: ntpd: command not found 2024-08-16T02:33:14.314 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-16T02:33:14.322 INFO:teuthology.orchestra.run.smithi133.stdout:506 Cannot talk to daemon 2024-08-16T02:33:14.337 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-16T02:33:14.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-08-16T02:33:14.349 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-08-16T02:33:14.351 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-16T02:33:14.363 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-16T02:33:14.377 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-16T02:33:14.404 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:33:14.407 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:33:14.407 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2024-08-16T02:33:14.407 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:14.407 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:14.407 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:14.407 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:14.442 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:33:14.445 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:33:14.446 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-08-16T02:33:14.446 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:14.446 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:14.446 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:14.446 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:14.447 INFO:teuthology.run_tasks:Running task install... 2024-08-16T02:33:14.455 DEBUG:teuthology.task.install:project ceph 2024-08-16T02:33:14.455 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}} 2024-08-16T02:33:14.455 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'} 2024-08-16T02:33:14.455 INFO:teuthology.task.install:Using flavor: crimson 2024-08-16T02:33:14.462 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-16T02:33:14.463 INFO:teuthology.task.install:extra packages: [] 2024-08-16T02:33:14.463 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'tag': None, 'wait_for_package': False} 2024-08-16T02:33:14.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:33:14.466 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'tag': None, 'wait_for_package': False} 2024-08-16T02:33:14.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:33:14.616 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/ 2024-08-16T02:33:14.616 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5812.g6f386a3f 2024-08-16T02:33:14.632 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/ 2024-08-16T02:33:14.633 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5812.g6f386a3f 2024-08-16T02:33:14.725 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-16T02:33:14.725 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:33:14.726 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-16T02:33:14.738 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-16T02:33:14.738 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:33:14.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-16T02:33:14.752 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-16T02:33:14.752 DEBUG:teuthology.orchestra.run.smithi133:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-16T02:33:14.763 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-16T02:33:14.763 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-16T02:33:14.817 DEBUG:teuthology.orchestra.run.smithi133:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-16T02:33:14.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-16T02:33:14.893 DEBUG:teuthology.orchestra.run.smithi133:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-16T02:33:14.903 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-16T02:33:14.956 INFO:teuthology.orchestra.run.smithi133.stdout:check_obsoletes = 1 2024-08-16T02:33:14.958 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum clean all 2024-08-16T02:33:14.966 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-08-16T02:33:14.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-08-16T02:33:15.191 INFO:teuthology.orchestra.run.smithi133.stdout:45 files removed 2024-08-16T02:33:15.206 INFO:teuthology.orchestra.run.smithi137.stdout:45 files removed 2024-08-16T02:33:15.219 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw 2024-08-16T02:33:15.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-08-16T02:33:15.823 INFO:teuthology.orchestra.run.smithi133.stdout:ceph packages for x86_64 276 kB/s | 88 kB 00:00 2024-08-16T02:33:15.845 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 268 kB/s | 88 kB 00:00 2024-08-16T02:33:16.173 INFO:teuthology.orchestra.run.smithi133.stdout:ceph noarch packages 80 kB/s | 24 kB 00:00 2024-08-16T02:33:16.203 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 78 kB/s | 24 kB 00:00 2024-08-16T02:33:16.507 INFO:teuthology.orchestra.run.smithi133.stdout:ceph source packages 7.2 kB/s | 2.1 kB 00:00 2024-08-16T02:33:16.546 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 7.0 kB/s | 2.1 kB 00:00 2024-08-16T02:33:17.282 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-08-16T02:33:17.862 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.2 MB 00:01 2024-08-16T02:33:19.175 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-08-16T02:33:21.172 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 20 MB 00:02 2024-08-16T02:33:23.667 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.6 MB 00:00 2024-08-16T02:33:25.558 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-16T02:33:25.568 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 18 kB 00:00 2024-08-16T02:33:26.681 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 23 MB/s | 22 MB 00:00 2024-08-16T02:33:27.415 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 18 kB 00:00 2024-08-16T02:33:28.894 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 17 MB/s | 22 MB 00:01 2024-08-16T02:33:32.137 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-08-16T02:33:34.358 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-08-16T02:33:34.547 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 10 M 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrading: 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.8 M 2024-08-16T02:33:34.564 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.1 M 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 20 M 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 25 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 807 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 558 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-16T02:33:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.4 M 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 45 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 138 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 185 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 364 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 357 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 117 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-16T02:33:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout:Installing weak dependencies: 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout:Install 30 Packages 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrade 2 Packages 2024-08-16T02:33:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:33:34.568 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 60 M 2024-08-16T02:33:34.568 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:33:35.371 INFO:teuthology.orchestra.run.smithi133.stdout:(1/32): ceph-base-19.0.0-5812.g6f386a3f.el9.x86 9.4 MB/s | 6.1 MB 00:00 2024-08-16T02:33:35.438 INFO:teuthology.orchestra.run.smithi133.stdout:(2/32): ceph-selinux-19.0.0-5812.g6f386a3f.el9. 377 kB/s | 25 kB 00:00 2024-08-16T02:33:35.522 INFO:teuthology.orchestra.run.smithi133.stdout:(3/32): libcephfs2-19.0.0-5812.g6f386a3f.el9.x8 9.5 MB/s | 807 kB 00:00 2024-08-16T02:33:35.638 INFO:teuthology.orchestra.run.smithi133.stdout:(4/32): ceph-radosgw-19.0.0-5812.g6f386a3f.el9. 11 MB/s | 10 MB 00:00 2024-08-16T02:33:35.680 INFO:teuthology.orchestra.run.smithi133.stdout:(5/32): libradosstriper1-19.0.0-5812.g6f386a3f. 3.4 MB/s | 558 kB 00:00 2024-08-16T02:33:35.747 INFO:teuthology.orchestra.run.smithi133.stdout:(6/32): python3-ceph-argparse-19.0.0-5812.g6f38 679 kB/s | 45 kB 00:00 2024-08-16T02:33:35.939 INFO:teuthology.orchestra.run.smithi133.stdout:(7/32): ceph-common-19.0.0-5812.g6f386a3f.el9.x 16 MB/s | 20 MB 00:01 2024-08-16T02:33:35.964 INFO:teuthology.orchestra.run.smithi133.stdout:(8/32): python3-ceph-common-19.0.0-5812.g6f386a 638 kB/s | 138 kB 00:00 2024-08-16T02:33:35.990 INFO:teuthology.orchestra.run.smithi133.stdout:(9/32): python3-cephfs-19.0.0-5812.g6f386a3f.el 3.6 MB/s | 185 kB 00:00 2024-08-16T02:33:36.015 INFO:teuthology.orchestra.run.smithi133.stdout:(10/32): python3-rados-19.0.0-5812.g6f386a3f.el 7.1 MB/s | 364 kB 00:00 2024-08-16T02:33:36.040 INFO:teuthology.orchestra.run.smithi133.stdout:(11/32): python3-rbd-19.0.0-5812.g6f386a3f.el9. 6.9 MB/s | 357 kB 00:00 2024-08-16T02:33:36.066 INFO:teuthology.orchestra.run.smithi133.stdout:(12/32): python3-rgw-19.0.0-5812.g6f386a3f.el9. 2.3 MB/s | 117 kB 00:00 2024-08-16T02:33:36.149 INFO:teuthology.orchestra.run.smithi133.stdout:(13/32): librgw2-19.0.0-5812.g6f386a3f.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-08-16T02:33:36.758 INFO:teuthology.orchestra.run.smithi133.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 149 kB/s | 107 kB 00:00 2024-08-16T02:33:36.765 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 10 M 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.8 M 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:33:36.776 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.1 M 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 20 M 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 25 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 807 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 558 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-16T02:33:36.777 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.4 M 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 45 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 138 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 185 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 364 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 357 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 117 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-16T02:33:36.778 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout:Install 30 Packages 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-08-16T02:33:36.779 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:33:36.780 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 60 M 2024-08-16T02:33:36.780 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:33:36.791 INFO:teuthology.orchestra.run.smithi133.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 70 kB/s | 45 kB 00:00 2024-08-16T02:33:36.825 INFO:teuthology.orchestra.run.smithi133.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 211 kB/s | 160 kB 00:00 2024-08-16T02:33:37.417 INFO:teuthology.orchestra.run.smithi133.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 71 kB/s | 42 kB 00:00 2024-08-16T02:33:37.584 INFO:teuthology.orchestra.run.smithi133.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 369 kB/s | 292 kB 00:00 2024-08-16T02:33:37.634 INFO:teuthology.orchestra.run.smithi133.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 757 kB/s | 662 kB 00:00 2024-08-16T02:33:38.193 INFO:teuthology.orchestra.run.smithi133.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 59 kB/s | 36 kB 00:00 2024-08-16T02:33:38.218 INFO:teuthology.orchestra.run.smithi133.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 276 kB/s | 221 kB 00:00 2024-08-16T02:33:38.235 INFO:teuthology.orchestra.run.smithi133.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 79 kB/s | 47 kB 00:00 2024-08-16T02:33:38.352 INFO:teuthology.orchestra.run.smithi133.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-08-16T02:33:38.369 INFO:teuthology.orchestra.run.smithi133.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.3 MB/s | 49 kB 00:00 2024-08-16T02:33:38.386 INFO:teuthology.orchestra.run.smithi133.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 265 kB/s | 26 kB 00:00 2024-08-16T02:33:38.411 INFO:teuthology.orchestra.run.smithi133.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-08-16T02:33:38.445 INFO:teuthology.orchestra.run.smithi133.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-08-16T02:33:38.459 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-radosgw-19.0.0-5812.g6f386a3f.el9. 9.7 MB/s | 10 MB 00:01 2024-08-16T02:33:38.495 INFO:teuthology.orchestra.run.smithi133.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-08-16T02:33:38.501 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-selinux-19.0.0-5812.g6f386a3f.el9. 604 kB/s | 25 kB 00:00 2024-08-16T02:33:38.562 INFO:teuthology.orchestra.run.smithi133.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-08-16T02:33:38.585 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): libcephfs2-19.0.0-5812.g6f386a3f.el9.x8 9.5 MB/s | 807 kB 00:00 2024-08-16T02:33:38.646 INFO:teuthology.orchestra.run.smithi133.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-08-16T02:33:38.777 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): ceph-common-19.0.0-5812.g6f386a3f.el9.x 14 MB/s | 20 MB 00:01 2024-08-16T02:33:38.804 INFO:teuthology.orchestra.run.smithi133.stdout:(31/32): librados2-19.0.0-5812.g6f386a3f.el9.x8 11 MB/s | 3.5 MB 00:00 2024-08-16T02:33:38.810 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): libradosstriper1-19.0.0-5812.g6f386a3f. 2.4 MB/s | 558 kB 00:00 2024-08-16T02:33:38.869 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): python3-ceph-argparse-19.0.0-5812.g6f38 772 kB/s | 45 kB 00:00 2024-08-16T02:33:38.905 INFO:teuthology.orchestra.run.smithi133.stdout:(32/32): librbd1-19.0.0-5812.g6f386a3f.el9.x86_ 11 MB/s | 3.8 MB 00:00 2024-08-16T02:33:38.909 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:33:38.910 INFO:teuthology.orchestra.run.smithi133.stdout:Total 14 MB/s | 60 MB 00:04 2024-08-16T02:33:38.928 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): python3-ceph-common-19.0.0-5812.g6f386a 2.3 MB/s | 138 kB 00:00 2024-08-16T02:33:38.987 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): python3-cephfs-19.0.0-5812.g6f386a3f.el 3.1 MB/s | 185 kB 00:00 2024-08-16T02:33:39.010 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:33:39.045 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:33:39.098 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:33:39.099 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): ceph-base-19.0.0-5812.g6f386a3f.el9.x86 3.6 MB/s | 6.1 MB 00:01 2024-08-16T02:33:39.162 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): librgw2-19.0.0-5812.g6f386a3f.el9.x86_ 14 MB/s | 5.4 MB 00:00 2024-08-16T02:33:39.187 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-rados-19.0.0-5812.g6f386a3f.el 1.8 MB/s | 364 kB 00:00 2024-08-16T02:33:39.213 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): python3-rgw-19.0.0-5812.g6f386a3f.el9. 2.3 MB/s | 117 kB 00:00 2024-08-16T02:33:39.238 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): python3-rbd-19.0.0-5812.g6f386a3f.el9. 2.3 MB/s | 357 kB 00:00 2024-08-16T02:33:39.380 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 556 kB/s | 107 kB 00:00 2024-08-16T02:33:39.405 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 271 kB/s | 45 kB 00:00 2024-08-16T02:33:39.430 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 737 kB/s | 160 kB 00:00 2024-08-16T02:33:39.489 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 723 kB/s | 42 kB 00:00 2024-08-16T02:33:39.531 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-08-16T02:33:39.556 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-08-16T02:33:39.590 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-08-16T02:33:39.615 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 428 kB/s | 36 kB 00:00 2024-08-16T02:33:39.617 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:33:39.618 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:33:39.640 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-08-16T02:33:39.757 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-08-16T02:33:39.783 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-16T02:33:39.808 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 229 kB/s | 26 kB 00:00 2024-08-16T02:33:39.825 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-08-16T02:33:39.850 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-08-16T02:33:39.900 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-08-16T02:33:39.951 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-16T02:33:40.026 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-08-16T02:33:40.201 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): librbd1-19.0.0-5812.g6f386a3f.el9.x86_ 15 MB/s | 3.8 MB 00:00 2024-08-16T02:33:40.318 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): librados2-19.0.0-5812.g6f386a3f.el9.x8 8.3 MB/s | 3.5 MB 00:00 2024-08-16T02:33:40.323 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:33:40.323 INFO:teuthology.orchestra.run.smithi137.stdout:Total 17 MB/s | 60 MB 00:03 2024-08-16T02:33:40.422 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:33:40.457 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:33:40.457 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:33:41.019 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:33:41.024 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:33:41.024 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:33:41.522 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-16T02:33:41.573 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:33:41.670 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:33:41.769 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:33:41.808 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:33:41.871 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:33:41.917 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-08-16T02:33:41.946 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-08-16T02:33:41.971 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-08-16T02:33:41.994 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:33:42.019 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:33:42.204 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:33:42.235 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:33:42.421 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:33:42.600 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-08-16T02:33:42.665 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:33:42.715 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:33:42.759 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 12/34 2024-08-16T02:33:42.818 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:33:42.845 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 14/34 2024-08-16T02:33:42.914 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-16T02:33:42.924 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-08-16T02:33:42.959 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:33:43.020 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:33:43.032 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:33:43.075 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-08-16T02:33:43.124 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:33:43.186 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-08-16T02:33:43.217 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 19/34 2024-08-16T02:33:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:33:43.258 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-08-16T02:33:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:33:43.290 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-08-16T02:33:43.319 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 22/34 2024-08-16T02:33:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:33:43.350 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-08-16T02:33:43.363 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-08-16T02:33:43.390 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-16T02:33:43.393 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-08-16T02:33:43.419 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-08-16T02:33:43.441 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:33:43.460 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:33:43.651 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:33:43.682 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:33:43.978 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-16T02:33:44.015 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:33:44.043 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-08-16T02:33:44.071 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:33:44.095 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:33:44.139 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 27/34 2024-08-16T02:33:44.145 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:33:44.168 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 28/34 2024-08-16T02:33:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 12/34 2024-08-16T02:33:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:33:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 14/34 2024-08-16T02:33:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-08-16T02:33:44.389 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:33:44.476 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:33:44.513 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-08-16T02:33:44.623 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-08-16T02:33:44.647 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 19/34 2024-08-16T02:33:44.688 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-08-16T02:33:44.720 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-08-16T02:33:44.749 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 22/34 2024-08-16T02:33:44.781 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-08-16T02:33:44.817 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-16T02:33:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-16T02:33:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:33:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:33:45.568 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 27/34 2024-08-16T02:33:45.598 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 28/34 2024-08-16T02:33:46.672 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:33:46.714 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:33:47.417 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:33:47.460 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:33:47.557 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:33:47.557 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-16T02:33:47.557 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-16T02:33:47.558 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:33:47.577 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:33:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:33:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:33:48.871 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:33:48.916 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:33:49.019 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:33:49.019 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-16T02:33:49.019 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-16T02:33:49.020 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:33:49.058 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:33:57.744 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:33:57.744 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /sys 2024-08-16T02:33:57.744 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /proc 2024-08-16T02:33:57.744 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /mnt 2024-08-16T02:33:57.744 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /var/tmp 2024-08-16T02:33:57.744 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /home 2024-08-16T02:33:57.744 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /root 2024-08-16T02:33:57.745 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /tmp 2024-08-16T02:33:57.745 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:33:58.819 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:33:58.844 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:33:58.844 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:33:58.845 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-16T02:33:58.845 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:33:58.845 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:33:58.845 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:33:58.846 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:33:58.930 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:33:58.930 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:33:59.336 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:33:59.336 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-08-16T02:33:59.336 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-08-16T02:33:59.336 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-08-16T02:33:59.336 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-08-16T02:33:59.336 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-08-16T02:33:59.337 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-08-16T02:33:59.337 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-08-16T02:33:59.337 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:00.143 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:00.143 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/34 2024-08-16T02:34:00.143 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:34:00.143 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 6/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 7/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 8/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 9/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 10/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 12/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:34:00.144 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-16T02:34:00.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-16T02:34:00.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-16T02:34:00.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-16T02:34:00.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-16T02:34:00.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:34:00.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-16T02:34:00.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/34 2024-08-16T02:34:00.429 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:34:00.453 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:34:00.453 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:00.453 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-16T02:34:00.453 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:34:00.454 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:34:00.454 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:00.455 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:34:00.539 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:34:00.539 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:01.107 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:01.107 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:01.107 INFO:teuthology.orchestra.run.smithi133.stdout:Upgraded: 2024-08-16T02:34:01.107 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-16T02:34:01.108 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-16T02:34:01.109 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-16T02:34:01.110 INFO:teuthology.orchestra.run.smithi133.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-16T02:34:01.110 INFO:teuthology.orchestra.run.smithi133.stdout: re2-1:20211101-3.el9.x86_64 2024-08-16T02:34:01.110 INFO:teuthology.orchestra.run.smithi133.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-16T02:34:01.110 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:01.110 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:34:01.459 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-test 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 6/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 7/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 8/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 9/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 10/34 2024-08-16T02:34:01.813 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 12/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-16T02:34:01.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-16T02:34:01.815 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/34 2024-08-16T02:34:02.019 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:30 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:34:02.147 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:34:02.148 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:02.148 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:34:02.148 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:02.148 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:34:02.148 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 86 M 2024-08-16T02:34:02.148 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: libasan8 x86_64 13.3.1-2.1.el9 appstream 469 k 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 175 k 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:02.149 INFO:teuthology.orchestra.run.smithi133.stdout:Install 9 Packages 2024-08-16T02:34:02.150 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:02.150 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 88 M 2024-08-16T02:34:02.150 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 834 M 2024-08-16T02:34:02.150 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:34:02.626 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:02.626 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:02.626 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-08-16T02:34:02.626 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.626 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-16T02:34:02.627 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-16T02:34:02.628 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-16T02:34:02.629 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-16T02:34:02.629 INFO:teuthology.orchestra.run.smithi137.stdout: re2-1:20211101-3.el9.x86_64 2024-08-16T02:34:02.629 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-16T02:34:02.629 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:02.629 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:34:02.919 INFO:teuthology.orchestra.run.smithi133.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 444 kB/s | 94 kB 00:00 2024-08-16T02:34:02.944 INFO:teuthology.orchestra.run.smithi133.stdout:(2/9): libcephsqlite-19.0.0-5812.g6f386a3f.el9. 736 kB/s | 175 kB 00:00 2024-08-16T02:34:02.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-08-16T02:34:03.036 INFO:teuthology.orchestra.run.smithi133.stdout:(3/9): libasan8-13.3.1-2.1.el9.x86_64.rpm 3.9 MB/s | 469 kB 00:00 2024-08-16T02:34:03.137 INFO:teuthology.orchestra.run.smithi133.stdout:(4/9): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-08-16T02:34:03.162 INFO:teuthology.orchestra.run.smithi133.stdout:(5/9): libubsan-11.4.1-3.el9.x86_64.rpm 860 kB/s | 187 kB 00:00 2024-08-16T02:34:03.212 INFO:teuthology.orchestra.run.smithi133.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-08-16T02:34:03.262 INFO:teuthology.orchestra.run.smithi133.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.4 MB/s | 122 kB 00:00 2024-08-16T02:34:03.304 INFO:teuthology.orchestra.run.smithi133.stdout:(8/9): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-08-16T02:34:03.528 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:29 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:34:03.655 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 86 M 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout: libasan8 x86_64 13.3.1-2.1.el9 appstream 469 k 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 175 k 2024-08-16T02:34:03.656 INFO:teuthology.orchestra.run.smithi137.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout:Install 9 Packages 2024-08-16T02:34:03.657 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:03.658 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 88 M 2024-08-16T02:34:03.658 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 834 M 2024-08-16T02:34:03.658 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:34:04.193 INFO:teuthology.orchestra.run.smithi137.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 657 kB/s | 94 kB 00:00 2024-08-16T02:34:04.419 INFO:teuthology.orchestra.run.smithi137.stdout:(2/9): libasan8-13.3.1-2.1.el9.x86_64.rpm 2.0 MB/s | 469 kB 00:00 2024-08-16T02:34:04.536 INFO:teuthology.orchestra.run.smithi137.stdout:(3/9): libcephsqlite-19.0.0-5812.g6f386a3f.el9. 360 kB/s | 175 kB 00:00 2024-08-16T02:34:04.611 INFO:teuthology.orchestra.run.smithi137.stdout:(4/9): libubsan-11.4.1-3.el9.x86_64.rpm 972 kB/s | 187 kB 00:00 2024-08-16T02:34:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-08-16T02:34:04.745 INFO:teuthology.orchestra.run.smithi137.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-08-16T02:34:04.804 INFO:teuthology.orchestra.run.smithi137.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-08-16T02:34:04.862 INFO:teuthology.orchestra.run.smithi137.stdout:(8/9): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-08-16T02:34:05.246 INFO:teuthology.orchestra.run.smithi133.stdout:(9/9): ceph-test-19.0.0-5812.g6f386a3f.el9.x86_ 34 MB/s | 86 MB 00:02 2024-08-16T02:34:05.249 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:05.249 INFO:teuthology.orchestra.run.smithi133.stdout:Total 29 MB/s | 88 MB 00:03 2024-08-16T02:34:05.332 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:34:05.355 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:34:05.355 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:34:05.965 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:34:05.966 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:34:06.877 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:34:06.957 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-08-16T02:34:06.997 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-08-16T02:34:07.042 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-08-16T02:34:07.068 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-08-16T02:34:07.098 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:34:07.124 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:34:07.198 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:34:07.237 INFO:teuthology.orchestra.run.smithi137.stdout:(9/9): ceph-test-19.0.0-5812.g6f386a3f.el9.x86_ 27 MB/s | 86 MB 00:03 2024-08-16T02:34:07.240 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:07.241 INFO:teuthology.orchestra.run.smithi137.stdout:Total 25 MB/s | 88 MB 00:03 2024-08-16T02:34:07.250 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-08-16T02:34:07.276 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:34:07.323 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:34:07.345 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:34:07.345 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:34:07.949 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:34:07.950 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:34:08.854 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:34:08.923 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-08-16T02:34:08.965 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-08-16T02:34:09.008 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-08-16T02:34:09.035 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-08-16T02:34:09.065 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:34:09.090 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:34:09.157 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:34:09.200 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-08-16T02:34:09.226 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:34:19.038 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:34:19.124 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:34:20.838 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:34:20.924 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:34:25.511 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:34:25.511 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/9 2024-08-16T02:34:25.511 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/9 2024-08-16T02:34:25.512 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-08-16T02:34:25.512 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libasan8-13.3.1-2.1.el9.x86_64 4/9 2024-08-16T02:34:25.512 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:34:25.512 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-08-16T02:34:25.512 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-08-16T02:34:25.512 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-08-16T02:34:25.828 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-08-16T02:34:25.828 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:25.828 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:34:25.828 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:25.828 INFO:teuthology.orchestra.run.smithi133.stdout: libasan8-13.3.1-2.1.el9.x86_64 2024-08-16T02:34:25.828 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:25.828 INFO:teuthology.orchestra.run.smithi133.stdout: libubsan-11.4.1-3.el9.x86_64 2024-08-16T02:34:25.829 INFO:teuthology.orchestra.run.smithi133.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-08-16T02:34:25.829 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-16T02:34:25.829 INFO:teuthology.orchestra.run.smithi133.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-16T02:34:25.829 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-16T02:34:25.829 INFO:teuthology.orchestra.run.smithi133.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-08-16T02:34:25.829 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:25.829 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:34:26.121 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph 2024-08-16T02:34:26.680 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:54 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:34:26.812 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:34:26.814 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:26.815 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:34:26.815 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:26.815 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:34:26.815 INFO:teuthology.orchestra.run.smithi133.stdout: ceph x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.5 k 2024-08-16T02:34:26.815 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:34:26.815 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 2.2 M 2024-08-16T02:34:26.815 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 1.6 M 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 284 k 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.1 M 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 72 M 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-16T02:34:26.816 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-16T02:34:26.817 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:26.818 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:34:26.819 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:26.819 INFO:teuthology.orchestra.run.smithi133.stdout:Install 37 Packages 2024-08-16T02:34:26.819 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:26.819 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 86 M 2024-08-16T02:34:26.820 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 943 M 2024-08-16T02:34:26.820 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:34:27.412 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:34:27.412 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/9 2024-08-16T02:34:27.413 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/9 2024-08-16T02:34:27.413 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-08-16T02:34:27.413 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libasan8-13.3.1-2.1.el9.x86_64 4/9 2024-08-16T02:34:27.413 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:34:27.413 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-08-16T02:34:27.413 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-08-16T02:34:27.413 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-08-16T02:34:27.595 INFO:teuthology.orchestra.run.smithi133.stdout:(1/37): ceph-19.0.0-5812.g6f386a3f.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-08-16T02:34:27.747 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-08-16T02:34:27.747 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:27.747 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:34:27.747 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:27.747 INFO:teuthology.orchestra.run.smithi137.stdout: libasan8-13.3.1-2.1.el9.x86_64 2024-08-16T02:34:27.747 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:27.747 INFO:teuthology.orchestra.run.smithi137.stdout: libubsan-11.4.1-3.el9.x86_64 2024-08-16T02:34:27.747 INFO:teuthology.orchestra.run.smithi137.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-08-16T02:34:27.748 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-16T02:34:27.748 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-16T02:34:27.748 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-16T02:34:27.748 INFO:teuthology.orchestra.run.smithi137.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-08-16T02:34:27.748 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:27.748 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:34:27.812 INFO:teuthology.orchestra.run.smithi133.stdout:(2/37): ceph-mds-19.0.0-5812.g6f386a3f.el9.x86_ 5.7 MB/s | 2.2 MB 00:00 2024-08-16T02:34:27.854 INFO:teuthology.orchestra.run.smithi133.stdout:(3/37): ceph-mgr-19.0.0-5812.g6f386a3f.el9.x86_ 3.8 MB/s | 1.6 MB 00:00 2024-08-16T02:34:27.938 INFO:teuthology.orchestra.run.smithi133.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5812.g6f38 3.3 MB/s | 284 kB 00:00 2024-08-16T02:34:28.013 INFO:teuthology.orchestra.run.smithi133.stdout:(5/37): ceph-mon-19.0.0-5812.g6f386a3f.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-08-16T02:34:28.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-08-16T02:34:28.616 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:54 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:34:28.721 INFO:teuthology.orchestra.run.smithi133.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 323 kB/s | 253 kB 00:00 2024-08-16T02:34:28.744 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:34:28.747 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:28.747 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:34:28.747 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:28.747 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:34:28.747 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.5 k 2024-08-16T02:34:28.747 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:34:28.747 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 2.2 M 2024-08-16T02:34:28.747 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 1.6 M 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 284 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.1 M 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 72 M 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-16T02:34:28.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-16T02:34:28.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:28.750 INFO:teuthology.orchestra.run.smithi137.stdout:Install 37 Packages 2024-08-16T02:34:28.751 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:28.751 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 86 M 2024-08-16T02:34:28.751 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 943 M 2024-08-16T02:34:28.751 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:34:28.947 INFO:teuthology.orchestra.run.smithi133.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 1.3 MB/s | 1.2 MB 00:00 2024-08-16T02:34:29.466 INFO:teuthology.orchestra.run.smithi137.stdout:(1/37): ceph-19.0.0-5812.g6f386a3f.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-08-16T02:34:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:(2/37): ceph-mds-19.0.0-5812.g6f386a3f.el9.x86_ 5.8 MB/s | 2.2 MB 00:00 2024-08-16T02:34:29.717 INFO:teuthology.orchestra.run.smithi137.stdout:(3/37): ceph-mgr-19.0.0-5812.g6f386a3f.el9.x86_ 3.8 MB/s | 1.6 MB 00:00 2024-08-16T02:34:29.756 INFO:teuthology.orchestra.run.smithi133.stdout:(8/37): ceph-osd-19.0.0-5812.g6f386a3f.el9.x86_ 37 MB/s | 72 MB 00:01 2024-08-16T02:34:29.817 INFO:teuthology.orchestra.run.smithi137.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5812.g6f38 2.8 MB/s | 284 kB 00:00 2024-08-16T02:34:29.884 INFO:teuthology.orchestra.run.smithi137.stdout:(5/37): ceph-mon-19.0.0-5812.g6f386a3f.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-08-16T02:34:29.897 INFO:teuthology.orchestra.run.smithi133.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 91 kB/s | 106 kB 00:01 2024-08-16T02:34:30.084 INFO:teuthology.orchestra.run.smithi137.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 949 kB/s | 253 kB 00:00 2024-08-16T02:34:30.143 INFO:teuthology.orchestra.run.smithi137.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-08-16T02:34:30.185 INFO:teuthology.orchestra.run.smithi137.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 4.1 MB/s | 1.2 MB 00:00 2024-08-16T02:34:30.210 INFO:teuthology.orchestra.run.smithi137.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-08-16T02:34:30.243 INFO:teuthology.orchestra.run.smithi137.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-08-16T02:34:30.269 INFO:teuthology.orchestra.run.smithi137.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-08-16T02:34:30.356 INFO:teuthology.orchestra.run.smithi133.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 98 kB/s | 135 kB 00:01 2024-08-16T02:34:30.386 INFO:teuthology.orchestra.run.smithi137.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-08-16T02:34:30.411 INFO:teuthology.orchestra.run.smithi137.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-08-16T02:34:30.436 INFO:teuthology.orchestra.run.smithi137.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 691 kB/s | 35 kB 00:00 2024-08-16T02:34:30.453 INFO:teuthology.orchestra.run.smithi137.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-08-16T02:34:30.473 INFO:teuthology.orchestra.run.smithi133.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-08-16T02:34:30.478 INFO:teuthology.orchestra.run.smithi137.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-08-16T02:34:30.498 INFO:teuthology.orchestra.run.smithi133.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 175 kB/s | 125 kB 00:00 2024-08-16T02:34:30.503 INFO:teuthology.orchestra.run.smithi137.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 829 kB/s | 42 kB 00:00 2024-08-16T02:34:30.540 INFO:teuthology.orchestra.run.smithi133.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-08-16T02:34:30.554 INFO:teuthology.orchestra.run.smithi137.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-08-16T02:34:30.590 INFO:teuthology.orchestra.run.smithi133.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 377 kB/s | 35 kB 00:00 2024-08-16T02:34:30.604 INFO:teuthology.orchestra.run.smithi137.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-08-16T02:34:30.616 INFO:teuthology.orchestra.run.smithi133.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-08-16T02:34:30.621 INFO:teuthology.orchestra.run.smithi137.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 671 kB/s | 43 kB 00:00 2024-08-16T02:34:30.648 INFO:teuthology.orchestra.run.smithi137.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.2 MB/s | 172 kB 00:00 2024-08-16T02:34:30.657 INFO:teuthology.orchestra.run.smithi133.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 766 kB/s | 51 kB 00:00 2024-08-16T02:34:30.663 INFO:teuthology.orchestra.run.smithi137.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 644 kB/s | 11 kB 00:00 2024-08-16T02:34:30.682 INFO:teuthology.orchestra.run.smithi137.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-08-16T02:34:30.683 INFO:teuthology.orchestra.run.smithi133.stdout:(17/37): python3-urllib3-1.26.5-6.el9.noarch.rp 274 kB/s | 215 kB 00:00 2024-08-16T02:34:30.697 INFO:teuthology.orchestra.run.smithi137.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-08-16T02:34:30.708 INFO:teuthology.orchestra.run.smithi133.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 452 kB/s | 42 kB 00:00 2024-08-16T02:34:30.723 INFO:teuthology.orchestra.run.smithi137.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 774 kB/s | 19 kB 00:00 2024-08-16T02:34:30.748 INFO:teuthology.orchestra.run.smithi137.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-08-16T02:34:30.775 INFO:teuthology.orchestra.run.smithi137.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-08-16T02:34:30.791 INFO:teuthology.orchestra.run.smithi133.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-08-16T02:34:30.798 INFO:teuthology.orchestra.run.smithi137.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-08-16T02:34:30.817 INFO:teuthology.orchestra.run.smithi133.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-08-16T02:34:30.824 INFO:teuthology.orchestra.run.smithi137.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-16T02:34:30.842 INFO:teuthology.orchestra.run.smithi133.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 325 kB/s | 43 kB 00:00 2024-08-16T02:34:30.849 INFO:teuthology.orchestra.run.smithi137.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 659 kB/s | 16 kB 00:00 2024-08-16T02:34:30.883 INFO:teuthology.orchestra.run.smithi137.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-08-16T02:34:30.892 INFO:teuthology.orchestra.run.smithi133.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 407 kB/s | 11 kB 00:00 2024-08-16T02:34:30.908 INFO:teuthology.orchestra.run.smithi137.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-08-16T02:34:30.917 INFO:teuthology.orchestra.run.smithi133.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-08-16T02:34:30.934 INFO:teuthology.orchestra.run.smithi137.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-08-16T02:34:30.943 INFO:teuthology.orchestra.run.smithi133.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 370 kB/s | 18 kB 00:00 2024-08-16T02:34:30.959 INFO:teuthology.orchestra.run.smithi137.stdout:(34/37): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-08-16T02:34:30.968 INFO:teuthology.orchestra.run.smithi133.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-08-16T02:34:30.984 INFO:teuthology.orchestra.run.smithi137.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-08-16T02:34:30.993 INFO:teuthology.orchestra.run.smithi133.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-08-16T02:34:31.010 INFO:teuthology.orchestra.run.smithi137.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-08-16T02:34:31.019 INFO:teuthology.orchestra.run.smithi133.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-08-16T02:34:31.044 INFO:teuthology.orchestra.run.smithi133.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-08-16T02:34:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-08-16T02:34:31.086 INFO:teuthology.orchestra.run.smithi133.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 659 kB/s | 16 kB 00:00 2024-08-16T02:34:31.111 INFO:teuthology.orchestra.run.smithi133.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-08-16T02:34:31.137 INFO:teuthology.orchestra.run.smithi133.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-08-16T02:34:31.162 INFO:teuthology.orchestra.run.smithi133.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-08-16T02:34:31.188 INFO:teuthology.orchestra.run.smithi133.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-08-16T02:34:31.212 INFO:teuthology.orchestra.run.smithi133.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 810 kB/s | 20 kB 00:00 2024-08-16T02:34:31.238 INFO:teuthology.orchestra.run.smithi133.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-16T02:34:31.282 INFO:teuthology.orchestra.run.smithi133.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.5 MB/s | 427 kB 00:00 2024-08-16T02:34:31.286 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:31.286 INFO:teuthology.orchestra.run.smithi133.stdout:Total 19 MB/s | 86 MB 00:04 2024-08-16T02:34:31.431 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:34:31.472 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:34:31.472 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:34:32.235 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:34:32.236 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:34:32.826 INFO:teuthology.orchestra.run.smithi137.stdout:(37/37): ceph-osd-19.0.0-5812.g6f386a3f.el9.x86 23 MB/s | 72 MB 00:03 2024-08-16T02:34:32.830 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:32.830 INFO:teuthology.orchestra.run.smithi137.stdout:Total 21 MB/s | 86 MB 00:04 2024-08-16T02:34:32.977 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:34:33.019 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:34:33.019 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:34:33.802 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:34:33.802 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:34:33.912 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:34:33.966 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-16T02:34:33.999 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-16T02:34:34.027 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-16T02:34:34.091 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-16T02:34:34.151 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-16T02:34:34.184 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-16T02:34:34.278 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-16T02:34:34.354 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-16T02:34:34.403 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-16T02:34:34.437 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-16T02:34:34.467 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-16T02:34:34.502 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-16T02:34:34.535 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-16T02:34:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-16T02:34:34.611 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-16T02:34:34.645 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-16T02:34:34.694 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-16T02:34:34.726 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-16T02:34:34.799 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-16T02:34:34.939 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-16T02:34:35.026 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-16T02:34:35.087 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-16T02:34:35.125 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-16T02:34:35.185 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-16T02:34:35.238 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-16T02:34:35.427 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-16T02:34:35.478 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:34:35.479 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-16T02:34:35.537 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-16T02:34:35.558 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-16T02:34:35.575 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-16T02:34:35.606 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-16T02:34:35.647 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-16T02:34:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-16T02:34:35.686 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-16T02:34:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-16T02:34:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-16T02:34:35.816 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-16T02:34:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-16T02:34:35.935 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-16T02:34:35.983 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-16T02:34:36.015 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 32/37 2024-08-16T02:34:36.018 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-16T02:34:36.046 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:34:36.047 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-16T02:34:36.073 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:34:36.073 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:36.073 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-16T02:34:36.073 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:34:36.073 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:34:36.073 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:36.082 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-16T02:34:36.115 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-16T02:34:36.156 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-16T02:34:36.191 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-16T02:34:36.225 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-16T02:34:36.274 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-16T02:34:36.306 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-16T02:34:36.388 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-16T02:34:36.537 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-16T02:34:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-16T02:34:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-16T02:34:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-16T02:34:36.763 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-16T02:34:36.818 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-16T02:34:37.008 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-16T02:34:37.059 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-16T02:34:37.137 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-16T02:34:37.226 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-16T02:34:37.266 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-16T02:34:37.394 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-16T02:34:37.585 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 32/37 2024-08-16T02:34:37.618 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:34:37.643 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:34:37.643 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:37.643 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-16T02:34:37.643 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:34:37.644 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:34:37.644 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:47.096 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:34:47.123 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:34:47.123 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:47.123 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-16T02:34:47.123 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:34:47.123 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:34:47.123 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:47.688 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:34:47.712 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:34:47.712 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:47.712 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-16T02:34:47.712 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:34:47.712 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:34:47.713 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:47.956 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:34:47.980 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:34:47.980 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:47.980 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-16T02:34:47.980 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:34:47.980 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:34:47.980 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:48.044 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:34:48.660 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:34:48.687 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:34:48.687 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:48.687 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-16T02:34:48.687 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:34:48.687 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:34:48.687 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:49.227 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:34:49.251 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:34:49.251 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:49.251 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-16T02:34:49.251 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:34:49.251 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:34:49.252 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:49.478 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:34:49.501 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:34:49.501 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:49.501 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-16T02:34:49.501 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:34:49.502 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:34:49.502 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:49.567 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:34:54.725 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:34:54.725 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/37 2024-08-16T02:34:54.725 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/37 2024-08-16T02:34:54.725 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/37 2024-08-16T02:34:54.725 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 6/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-16T02:34:54.726 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-16T02:34:54.727 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-16T02:34:54.728 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-16T02:34:54.728 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-16T02:34:54.729 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-16T02:34:54.729 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-16T02:34:54.729 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-16T02:34:54.729 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-16T02:34:54.729 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-16T02:34:54.729 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-16T02:34:54.729 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-16T02:34:55.582 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-16T02:34:55.582 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: lua-5.4.4-4.el9.x86_64 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-16T02:34:55.583 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-16T02:34:55.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply-3.11-14.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-16T02:34:55.585 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-16T02:34:55.586 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-16T02:34:55.586 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-16T02:34:55.586 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:55.586 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:34:55.947 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base 2024-08-16T02:34:56.503 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:24 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:34:56.536 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:34:56.536 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/37 2024-08-16T02:34:56.536 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/37 2024-08-16T02:34:56.536 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/37 2024-08-16T02:34:56.536 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/37 2024-08-16T02:34:56.536 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/37 2024-08-16T02:34:56.536 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 6/37 2024-08-16T02:34:56.536 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-16T02:34:56.537 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-16T02:34:56.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-16T02:34:56.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-16T02:34:56.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-16T02:34:56.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-16T02:34:56.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-16T02:34:56.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-16T02:34:56.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-16T02:34:56.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-16T02:34:56.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-16T02:34:56.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-16T02:34:56.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-16T02:34:56.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-16T02:34:56.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-16T02:34:56.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-16T02:34:56.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-16T02:34:56.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-16T02:34:56.589 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:34:56.630 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:34:56.631 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:34:56.631 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:34:56.692 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install cephadm 2024-08-16T02:34:57.236 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:25 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:34:57.364 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:34:57.365 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:57.365 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:34:57.365 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:57.365 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:34:57.365 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 773 k 2024-08-16T02:34:57.365 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:57.365 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:34:57.366 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:57.366 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-08-16T02:34:57.366 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:57.366 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 773 k 2024-08-16T02:34:57.366 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 779 k 2024-08-16T02:34:57.366 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:34:57.534 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-16T02:34:57.534 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:57.534 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:34:57.534 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:57.534 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:57.534 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:57.534 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: lua-5.4.4-4.el9.x86_64 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-16T02:34:57.535 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-16T02:34:57.536 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:57.537 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:34:57.645 INFO:teuthology.orchestra.run.smithi133.stdout:cephadm-19.0.0-5812.g6f386a3f.el9.noarch.rpm 2.7 MB/s | 773 kB 00:00 2024-08-16T02:34:57.645 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:57.646 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.7 MB/s | 773 kB 00:00 2024-08-16T02:34:57.646 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:34:57.651 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:34:57.651 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:34:57.656 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:34:57.657 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:34:57.731 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:34:57.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-08-16T02:34:58.128 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:34:58.238 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:34:58.365 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:24 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:34:58.456 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:34:58.499 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:34:58.500 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:34:58.500 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:34:58.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-08-16T02:34:58.624 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:34:58.882 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:34:58.882 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:58.882 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:34:58.882 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:34:58.882 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:58.883 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:34:59.120 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-immutable-object-cache 2024-08-16T02:34:59.133 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:25 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:34:59.262 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:34:59.263 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:59.263 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:34:59.263 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:59.263 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:34:59.263 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 773 k 2024-08-16T02:34:59.263 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:59.263 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:34:59.264 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:34:59.264 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-08-16T02:34:59.264 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:34:59.264 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 773 k 2024-08-16T02:34:59.264 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 779 k 2024-08-16T02:34:59.264 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:34:59.568 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-19.0.0-5812.g6f386a3f.el9.noarch.rpm 2.5 MB/s | 773 kB 00:00 2024-08-16T02:34:59.569 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:59.569 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.5 MB/s | 773 kB 00:00 2024-08-16T02:34:59.570 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:34:59.575 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:34:59.575 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:34:59.581 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:34:59.581 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:34:59.653 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:34:59.679 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:27 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:34:59.815 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:34:59.815 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:59.815 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-08-16T02:34:59.816 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:59.816 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:34:59.816 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 158 k 2024-08-16T02:34:59.816 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:59.816 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:34:59.816 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:34:59.816 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-08-16T02:34:59.816 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:34:59.817 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 158 k 2024-08-16T02:34:59.817 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 644 k 2024-08-16T02:34:59.817 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:00.040 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-immutable-object-cache-19.0.0-5812.g6f386a 708 kB/s | 158 kB 00:00 2024-08-16T02:35:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:Total 703 kB/s | 158 kB 00:00 2024-08-16T02:35:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:00.051 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:00.051 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:00.109 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:00.110 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:00.137 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:00.222 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:00.261 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:00.325 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:00.350 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:00.350 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:00.350 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-16T02:35:00.350 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:00.601 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:00.904 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:00.904 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:00.905 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:00.905 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:00.905 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:00.905 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:01.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-08-16T02:35:01.307 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:01.307 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:01.307 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:01.307 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:01.307 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:01.308 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:01.573 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr 2024-08-16T02:35:01.671 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:27 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:01.806 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:01.806 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:01.806 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-08-16T02:35:01.806 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:01.806 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:01.806 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 158 k 2024-08-16T02:35:01.806 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:01.806 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:01.807 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:01.807 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-08-16T02:35:01.807 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:01.807 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 158 k 2024-08-16T02:35:01.807 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 644 k 2024-08-16T02:35:01.807 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:02.031 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-19.0.0-5812.g6f386a 704 kB/s | 158 kB 00:00 2024-08-16T02:35:02.031 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:02.032 INFO:teuthology.orchestra.run.smithi137.stdout:Total 699 kB/s | 158 kB 00:00 2024-08-16T02:35:02.032 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:02.041 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:02.041 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:02.097 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:02.098 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:02.130 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:30 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:02.219 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:02.262 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:02.263 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:35:02.263 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:02.320 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-dashboard 2024-08-16T02:35:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:02.333 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:02.333 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-16T02:35:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:02.869 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:30 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:02.997 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:02.998 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:02.998 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:35:02.998 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:02.998 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:02.998 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 14 M 2024-08-16T02:35:02.998 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 34 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 16 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:02.999 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:03.000 INFO:teuthology.orchestra.run.smithi133.stdout:Install 11 Packages 2024-08-16T02:35:03.000 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:03.000 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 18 M 2024-08-16T02:35:03.000 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 154 M 2024-08-16T02:35:03.000 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:03.338 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:03.338 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:03.338 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:03.338 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:03.339 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:03.339 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:03.427 INFO:teuthology.orchestra.run.smithi133.stdout:(1/11): ceph-prometheus-alerts-19.0.0-5812.g6f3 86 kB/s | 16 kB 00:00 2024-08-16T02:35:03.452 INFO:teuthology.orchestra.run.smithi133.stdout:(2/11): ceph-grafana-dashboards-19.0.0-5812.g6f 158 kB/s | 34 kB 00:00 2024-08-16T02:35:03.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-08-16T02:35:03.586 INFO:teuthology.orchestra.run.smithi133.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-08-16T02:35:03.645 INFO:teuthology.orchestra.run.smithi133.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-08-16T02:35:03.712 INFO:teuthology.orchestra.run.smithi133.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.3 MB/s | 548 kB 00:00 2024-08-16T02:35:03.729 INFO:teuthology.orchestra.run.smithi133.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 239 kB/s | 19 kB 00:00 2024-08-16T02:35:03.754 INFO:teuthology.orchestra.run.smithi133.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-08-16T02:35:03.771 INFO:teuthology.orchestra.run.smithi133.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-08-16T02:35:03.796 INFO:teuthology.orchestra.run.smithi133.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-08-16T02:35:03.863 INFO:teuthology.orchestra.run.smithi133.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 14 MB/s | 2.0 MB 00:00 2024-08-16T02:35:04.005 INFO:teuthology.orchestra.run.smithi133.stdout:(11/11): ceph-mgr-dashboard-19.0.0-5812.g6f386a 18 MB/s | 14 MB 00:00 2024-08-16T02:35:04.009 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:04.009 INFO:teuthology.orchestra.run.smithi133.stdout:Total 18 MB/s | 18 MB 00:01 2024-08-16T02:35:04.040 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:04.058 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:04.059 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:04.105 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:30 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:04.196 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:04.238 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:04.239 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:35:04.239 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:04.294 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:04.295 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:04.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-08-16T02:35:04.849 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:30 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:04.855 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:04.945 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-08-16T02:35:04.975 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:04.976 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:04.977 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:35:04.977 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:04.977 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:04.977 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 14 M 2024-08-16T02:35:04.977 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:35:04.977 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-16T02:35:04.978 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 34 k 2024-08-16T02:35:04.978 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 16 k 2024-08-16T02:35:04.978 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-16T02:35:04.978 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-16T02:35:04.978 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-16T02:35:04.979 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-16T02:35:04.979 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-16T02:35:04.979 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-16T02:35:04.979 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-16T02:35:04.979 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:04.979 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:04.980 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:04.980 INFO:teuthology.orchestra.run.smithi137.stdout:Install 11 Packages 2024-08-16T02:35:04.980 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:04.981 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 18 M 2024-08-16T02:35:04.981 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 154 M 2024-08-16T02:35:04.981 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:04.983 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-08-16T02:35:05.084 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-08-16T02:35:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-08-16T02:35:05.261 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-08-16T02:35:05.346 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-08-16T02:35:05.397 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-08-16T02:35:05.425 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:35:05.457 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 9/11 2024-08-16T02:35:05.651 INFO:teuthology.orchestra.run.smithi137.stdout:(1/11): ceph-grafana-dashboards-19.0.0-5812.g6f 163 kB/s | 34 kB 00:00 2024-08-16T02:35:05.677 INFO:teuthology.orchestra.run.smithi137.stdout:(2/11): ceph-prometheus-alerts-19.0.0-5812.g6f3 70 kB/s | 16 kB 00:00 2024-08-16T02:35:05.810 INFO:teuthology.orchestra.run.smithi137.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-08-16T02:35:05.894 INFO:teuthology.orchestra.run.smithi137.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 6.4 MB/s | 548 kB 00:00 2024-08-16T02:35:05.919 INFO:teuthology.orchestra.run.smithi137.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-08-16T02:35:05.962 INFO:teuthology.orchestra.run.smithi137.stdout:(6/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-08-16T02:35:06.053 INFO:teuthology.orchestra.run.smithi137.stdout:(7/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-08-16T02:35:06.079 INFO:teuthology.orchestra.run.smithi137.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-08-16T02:35:06.104 INFO:teuthology.orchestra.run.smithi137.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 614 kB/s | 31 kB 00:00 2024-08-16T02:35:06.129 INFO:teuthology.orchestra.run.smithi137.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-08-16T02:35:06.621 INFO:teuthology.orchestra.run.smithi137.stdout:(11/11): ceph-mgr-dashboard-19.0.0-5812.g6f386a 12 MB/s | 14 MB 00:01 2024-08-16T02:35:06.625 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:06.625 INFO:teuthology.orchestra.run.smithi137.stdout:Total 11 MB/s | 18 MB 00:01 2024-08-16T02:35:06.654 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:06.673 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:06.673 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:06.911 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:06.911 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:07.457 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:07.544 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-08-16T02:35:07.574 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-08-16T02:35:07.679 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-08-16T02:35:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-08-16T02:35:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-08-16T02:35:07.944 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-08-16T02:35:07.988 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-08-16T02:35:08.016 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:35:08.048 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 9/11 2024-08-16T02:35:08.678 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 10/11 2024-08-16T02:35:08.737 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:35:10.611 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:35:10.611 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 1/11 2024-08-16T02:35:10.611 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 2/11 2024-08-16T02:35:10.611 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 3/11 2024-08-16T02:35:10.611 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-08-16T02:35:10.611 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-08-16T02:35:10.611 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-08-16T02:35:10.612 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-08-16T02:35:10.612 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:35:10.612 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-08-16T02:35:10.612 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-08-16T02:35:10.922 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-08-16T02:35:10.922 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:10.922 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:10.922 INFO:teuthology.orchestra.run.smithi133.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-16T02:35:10.923 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-16T02:35:10.924 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:10.924 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:11.189 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-16T02:35:11.236 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 10/11 2024-08-16T02:35:11.286 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:35:11.738 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:39 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:11.866 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:11.867 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-08-16T02:35:11.867 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 7.4 M 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-16T02:35:11.868 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-16T02:35:11.869 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-16T02:35:11.869 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:11.869 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:11.869 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-08-16T02:35:11.869 INFO:teuthology.orchestra.run.smithi133.stdout:Install 12 Packages 2024-08-16T02:35:11.869 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:11.869 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 38 M 2024-08-16T02:35:11.870 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 209 M 2024-08-16T02:35:11.870 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:12.318 INFO:teuthology.orchestra.run.smithi133.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 893 kB/s | 195 kB 00:00 2024-08-16T02:35:12.377 INFO:teuthology.orchestra.run.smithi133.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-08-16T02:35:12.561 INFO:teuthology.orchestra.run.smithi133.stdout:(3/12): ceph-mgr-diskprediction-local-19.0.0-58 16 MB/s | 7.4 MB 00:00 2024-08-16T02:35:12.577 INFO:teuthology.orchestra.run.smithi133.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 33 kB 00:00 2024-08-16T02:35:12.753 INFO:teuthology.orchestra.run.smithi133.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 201 kB/s | 35 kB 00:00 2024-08-16T02:35:12.811 INFO:teuthology.orchestra.run.smithi133.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 73 kB/s | 18 kB 00:00 2024-08-16T02:35:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:35:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 1/11 2024-08-16T02:35:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 2/11 2024-08-16T02:35:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 3/11 2024-08-16T02:35:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-08-16T02:35:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-08-16T02:35:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-08-16T02:35:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-08-16T02:35:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:35:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-08-16T02:35:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-08-16T02:35:13.236 INFO:teuthology.orchestra.run.smithi133.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 574 kB/s | 244 kB 00:00 2024-08-16T02:35:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-08-16T02:35:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-16T02:35:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-16T02:35:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:13.267 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:13.412 INFO:teuthology.orchestra.run.smithi133.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-08-16T02:35:13.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-16T02:35:13.520 INFO:teuthology.orchestra.run.smithi133.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-08-16T02:35:14.043 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:14.171 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:14.172 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-08-16T02:35:14.172 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:35:14.172 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-08-16T02:35:14.172 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:14.172 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 7.4 M 2024-08-16T02:35:14.172 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:35:14.172 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-16T02:35:14.172 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-08-16T02:35:14.173 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-08-16T02:35:14.174 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:14.174 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 38 M 2024-08-16T02:35:14.174 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 209 M 2024-08-16T02:35:14.174 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:14.485 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-08-16T02:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-08-16T02:35:14.602 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-08-16T02:35:14.652 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-08-16T02:35:14.711 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-08-16T02:35:14.811 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-58 14 MB/s | 7.4 MB 00:00 2024-08-16T02:35:14.953 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.0 MB/s | 3.0 MB 00:00 2024-08-16T02:35:15.045 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-08-16T02:35:15.070 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 943 kB/s | 244 kB 00:00 2024-08-16T02:35:15.096 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.9 MB/s | 250 kB 00:00 2024-08-16T02:35:15.179 INFO:teuthology.orchestra.run.smithi133.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.6 MB/s | 5.1 MB 00:01 2024-08-16T02:35:15.187 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 5.1 MB 00:00 2024-08-16T02:35:15.337 INFO:teuthology.orchestra.run.smithi133.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 1.8 MB/s | 4.7 MB 00:02 2024-08-16T02:35:15.479 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-08-16T02:35:15.481 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:15.481 INFO:teuthology.orchestra.run.smithi137.stdout:Total 29 MB/s | 38 MB 00:01 2024-08-16T02:35:15.834 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:15.860 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:15.860 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:16.307 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:16.307 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:16.862 INFO:teuthology.orchestra.run.smithi133.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.9 MB/s | 16 MB 00:03 2024-08-16T02:35:16.863 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:16.863 INFO:teuthology.orchestra.run.smithi133.stdout:Total 7.6 MB/s | 38 MB 00:04 2024-08-16T02:35:17.063 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:17.140 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-16T02:35:17.177 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:35:17.213 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:17.239 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:17.239 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:17.279 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:35:17.312 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-16T02:35:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-16T02:35:17.511 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-16T02:35:17.586 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-16T02:35:17.685 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:17.686 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:18.239 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-16T02:35:18.366 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-16T02:35:18.438 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:18.501 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-16T02:35:18.535 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:35:18.634 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:35:18.670 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-16T02:35:18.850 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-16T02:35:18.886 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-16T02:35:18.963 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-16T02:35:19.621 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:35:19.625 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-16T02:35:19.731 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-16T02:35:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-16T02:35:20.677 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:35:20.981 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:35:22.030 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-16T02:35:22.051 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:35:23.152 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:35:23.152 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 1/12 2024-08-16T02:35:23.152 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:35:23.152 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:35:23.152 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-16T02:35:23.152 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-16T02:35:23.152 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-16T02:35:23.153 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-16T02:35:23.153 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-16T02:35:23.153 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-16T02:35:23.153 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:35:23.153 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-16T02:35:23.520 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-16T02:35:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-16T02:35:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-16T02:35:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:23.522 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:23.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-08-16T02:35:24.386 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:35:24.386 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 1/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:35:24.387 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-16T02:35:24.454 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:50 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:24.583 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:24.584 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:24.585 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:35:24.585 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:24.585 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:24.585 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 49 k 2024-08-16T02:35:24.585 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:35:24.585 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-16T02:35:24.586 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-16T02:35:24.586 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-16T02:35:24.586 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-16T02:35:24.586 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-16T02:35:24.586 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-16T02:35:24.586 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-16T02:35:24.586 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-16T02:35:24.587 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-16T02:35:24.587 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-16T02:35:24.587 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-16T02:35:24.587 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-16T02:35:24.587 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:24.587 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:24.588 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:24.588 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-08-16T02:35:24.588 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:24.590 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.2 M 2024-08-16T02:35:24.590 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-08-16T02:35:24.590 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-16T02:35:24.764 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-16T02:35:24.765 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-16T02:35:24.765 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-16T02:35:24.765 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-16T02:35:24.765 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-16T02:35:24.765 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-16T02:35:24.765 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-16T02:35:24.765 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:24.765 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:25.167 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 189 kB/s | 19 kB 00:00 2024-08-16T02:35:25.187 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-rook 2024-08-16T02:35:25.192 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 26 kB 00:00 2024-08-16T02:35:25.301 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): ceph-mgr-rook-19.0.0-5812.g6f386a3f.el9 209 kB/s | 49 kB 00:00 2024-08-16T02:35:25.326 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-08-16T02:35:25.352 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 999 kB/s | 159 kB 00:00 2024-08-16T02:35:25.377 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-08-16T02:35:25.461 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 297 kB/s | 32 kB 00:00 2024-08-16T02:35:25.486 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-08-16T02:35:25.511 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 128 kB/s | 14 kB 00:00 2024-08-16T02:35:25.561 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-google-auth-2.33.0-1.el9.noarc 2.2 MB/s | 220 kB 00:00 2024-08-16T02:35:25.587 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-08-16T02:35:25.621 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-08-16T02:35:25.662 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-08-16T02:35:25.665 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:25.665 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-08-16T02:35:25.728 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:25.742 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:25.743 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:25.764 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:53 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:25.828 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:25.829 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:25.902 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:25.903 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:25.903 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:35:25.903 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 49 k 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-16T02:35:25.904 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-16T02:35:25.905 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-16T02:35:25.905 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-16T02:35:25.905 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-16T02:35:25.905 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:25.905 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:25.905 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:25.905 INFO:teuthology.orchestra.run.smithi133.stdout:Install 13 Packages 2024-08-16T02:35:25.905 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:25.906 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 2.2 M 2024-08-16T02:35:25.906 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 26 M 2024-08-16T02:35:25.906 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:26.330 INFO:teuthology.orchestra.run.smithi133.stdout:(1/13): ceph-mgr-rook-19.0.0-5812.g6f386a3f.el9 263 kB/s | 49 kB 00:00 2024-08-16T02:35:26.335 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:26.422 INFO:teuthology.orchestra.run.smithi133.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 92 kB/s | 26 kB 00:00 2024-08-16T02:35:26.447 INFO:teuthology.orchestra.run.smithi133.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 63 kB/s | 19 kB 00:00 2024-08-16T02:35:26.501 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-16T02:35:26.560 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-16T02:35:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-16T02:35:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-16T02:35:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-16T02:35:26.772 INFO:teuthology.orchestra.run.smithi133.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 453 kB/s | 159 kB 00:00 2024-08-16T02:35:26.797 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-16T02:35:26.848 INFO:teuthology.orchestra.run.smithi133.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 429 kB/s | 222 kB 00:00 2024-08-16T02:35:26.873 INFO:teuthology.orchestra.run.smithi133.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 654 kB/s | 279 kB 00:00 2024-08-16T02:35:26.898 INFO:teuthology.orchestra.run.smithi133.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 427 kB/s | 54 kB 00:00 2024-08-16T02:35:26.899 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-16T02:35:26.955 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-16T02:35:26.999 INFO:teuthology.orchestra.run.smithi133.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 213 kB/s | 32 kB 00:00 2024-08-16T02:35:27.024 INFO:teuthology.orchestra.run.smithi133.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 93 kB/s | 14 kB 00:00 2024-08-16T02:35:27.050 INFO:teuthology.orchestra.run.smithi133.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-16T02:35:27.066 INFO:teuthology.orchestra.run.smithi133.stdout:(11/13): python3-google-auth-2.33.0-1.el9.noarc 1.8 MB/s | 220 kB 00:00 2024-08-16T02:35:27.092 INFO:teuthology.orchestra.run.smithi133.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-08-16T02:35:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.3 MB/s | 1.0 MB 00:00 2024-08-16T02:35:27.127 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:27.128 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-08-16T02:35:27.192 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:27.206 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:27.207 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:27.292 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:27.293 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:27.677 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-16T02:35:27.736 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-16T02:35:27.770 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-16T02:35:27.789 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:27.819 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-16T02:35:27.848 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:35:27.963 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-16T02:35:28.011 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-16T02:35:28.059 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-16T02:35:28.095 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-16T02:35:28.136 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-16T02:35:28.238 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-16T02:35:28.349 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-16T02:35:28.398 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-16T02:35:28.617 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:35:28.617 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 1/13 2024-08-16T02:35:28.617 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-16T02:35:28.618 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-16T02:35:28.968 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-16T02:35:28.968 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:28.968 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:28.968 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:28.968 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-16T02:35:28.968 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-16T02:35:28.968 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:28.969 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:29.117 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-16T02:35:29.161 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-16T02:35:29.195 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-16T02:35:29.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-08-16T02:35:29.243 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-16T02:35:29.265 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:35:29.783 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:55 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:29.913 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:29.913 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:29.913 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 150 k 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:29.914 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:29.915 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-08-16T02:35:29.915 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:29.915 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 7.0 M 2024-08-16T02:35:29.915 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 33 M 2024-08-16T02:35:29.915 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:30.116 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 1/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-16T02:35:30.117 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-16T02:35:30.443 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-16T02:35:30.443 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-16T02:35:30.444 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-16T02:35:30.445 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-16T02:35:30.445 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-16T02:35:30.445 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:30.445 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:30.482 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-08-16T02:35:30.557 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-08-16T02:35:30.583 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): ceph-mgr-cephadm-19.0.0-5812.g6f386a3f.e 590 kB/s | 150 kB 00:00 2024-08-16T02:35:30.608 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-08-16T02:35:30.667 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 891 kB/s | 74 kB 00:00 2024-08-16T02:35:30.717 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-cephadm 2024-08-16T02:35:31.275 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:59 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:31.403 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:31.404 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:31.404 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:35:31.404 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:31.404 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:31.404 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 150 k 2024-08-16T02:35:31.404 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:35:31.404 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2024-08-16T02:35:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:31.406 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 7.0 M 2024-08-16T02:35:31.406 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 33 M 2024-08-16T02:35:31.406 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:31.875 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-08-16T02:35:31.877 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:31.877 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.6 MB/s | 7.0 MB 00:01 2024-08-16T02:35:31.938 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-08-16T02:35:31.948 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:31.959 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:31.960 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:32.005 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-08-16T02:35:32.030 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-08-16T02:35:32.056 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): ceph-mgr-cephadm-19.0.0-5812.g6f386a3f.e 620 kB/s | 150 kB 00:00 2024-08-16T02:35:32.058 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:32.058 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:32.081 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-08-16T02:35:32.399 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:32.557 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-16T02:35:32.601 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-16T02:35:33.113 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-16T02:35:33.196 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-16T02:35:33.263 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-16T02:35:33.289 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.0 MB/s | 6.0 MB 00:01 2024-08-16T02:35:33.290 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:35:33.291 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:33.291 INFO:teuthology.orchestra.run.smithi133.stdout:Total 3.7 MB/s | 7.0 MB 00:01 2024-08-16T02:35:33.361 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:33.372 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:33.372 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:33.469 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:33.470 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:33.784 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:33.954 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-16T02:35:34.007 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-16T02:35:34.094 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:35:34.094 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/6 2024-08-16T02:35:34.094 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-16T02:35:34.094 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-16T02:35:34.094 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-16T02:35:34.094 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-16T02:35:34.358 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-16T02:35:34.358 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:34.358 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:34.358 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:34.358 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-16T02:35:34.358 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-16T02:35:34.358 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-16T02:35:34.359 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-16T02:35:34.359 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-16T02:35:34.359 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:34.359 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:34.526 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-16T02:35:34.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-08-16T02:35:34.618 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-16T02:35:34.694 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-16T02:35:34.721 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:35:35.155 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:01 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:35.288 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 883 k 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:35.289 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:35.290 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:35.290 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-08-16T02:35:35.290 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:35.290 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 963 k 2024-08-16T02:35:35.290 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 4.1 M 2024-08-16T02:35:35.290 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:35.510 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:35:35.510 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/6 2024-08-16T02:35:35.510 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-16T02:35:35.510 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-16T02:35:35.510 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-16T02:35:35.510 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-16T02:35:35.519 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 481 kB/s | 80 kB 00:00 2024-08-16T02:35:35.678 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-19.0.0-5812.g6f386a3f.el9.x86_ 2.7 MB/s | 883 kB 00:00 2024-08-16T02:35:35.678 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:35.679 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.4 MB/s | 963 kB 00:00 2024-08-16T02:35:35.690 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:35.714 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:35.714 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:35.794 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-16T02:35:35.794 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:35.794 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:35.794 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:35.794 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-16T02:35:35.794 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-16T02:35:35.794 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-16T02:35:35.794 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-16T02:35:35.795 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-16T02:35:35.795 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:35.795 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:35.795 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:35.795 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:35.975 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:36.057 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-fuse 2024-08-16T02:35:36.133 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-16T02:35:36.196 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:35:36.623 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:04 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:36.761 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:36.762 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:36.762 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:35:36.762 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:36.762 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:36.762 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 883 k 2024-08-16T02:35:36.762 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-08-16T02:35:36.762 INFO:teuthology.orchestra.run.smithi133.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-16T02:35:36.763 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:36.763 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:36.763 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:36.763 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2024-08-16T02:35:36.763 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:36.763 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 963 k 2024-08-16T02:35:36.763 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 4.1 M 2024-08-16T02:35:36.763 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:36.860 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:35:36.860 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/2 2024-08-16T02:35:37.043 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 499 kB/s | 80 kB 00:00 2024-08-16T02:35:37.151 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-16T02:35:37.151 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:37.152 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:37.152 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-16T02:35:37.152 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:37.152 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): ceph-fuse-19.0.0-5812.g6f386a3f.el9.x86_ 2.8 MB/s | 883 kB 00:00 2024-08-16T02:35:37.195 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:37.195 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.2 MB/s | 963 kB 00:00 2024-08-16T02:35:37.207 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:37.231 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:37.232 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:37.310 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:37.310 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:37.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-volume 2024-08-16T02:35:37.505 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:37.648 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-16T02:35:37.720 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:35:37.991 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:03 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:38.126 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 285 k 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 285 k 2024-08-16T02:35:38.127 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 1.3 M 2024-08-16T02:35:38.128 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:38.386 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:35:38.386 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/2 2024-08-16T02:35:38.389 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-volume-19.0.0-5812.g6f386a3f.el9.noarch.rp 1.1 MB/s | 285 kB 00:00 2024-08-16T02:35:38.390 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:38.390 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-08-16T02:35:38.390 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:38.398 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:38.398 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:38.428 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:38.428 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:38.648 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-16T02:35:38.648 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:38.648 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:38.648 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-16T02:35:38.648 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:38.648 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:38.652 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:38.716 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:38.732 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:38.732 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:38.732 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-16T02:35:38.732 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:38.884 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-volume 2024-08-16T02:35:39.454 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:07 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:39.587 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 285 k 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-08-16T02:35:39.588 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:39.589 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 285 k 2024-08-16T02:35:39.589 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.3 M 2024-08-16T02:35:39.589 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:39.739 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:39.739 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:39.740 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:39.740 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:39.740 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:39.740 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:39.836 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-volume-19.0.0-5812.g6f386a3f.el9.noarch.rp 1.1 MB/s | 285 kB 00:00 2024-08-16T02:35:39.836 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:39.837 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-08-16T02:35:39.837 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:39.845 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:39.845 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:39.878 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:39.879 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:40.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-08-16T02:35:40.097 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:40.171 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:40.187 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:40.188 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:40.188 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-16T02:35:40.188 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:40.563 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:06 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:40.693 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:40.693 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:40.693 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-08-16T02:35:40.693 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:40.693 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:40.693 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 125 k 2024-08-16T02:35:40.694 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:40.694 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:40.694 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:40.694 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-08-16T02:35:40.694 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:40.694 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 125 k 2024-08-16T02:35:40.695 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 445 k 2024-08-16T02:35:40.695 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:40.930 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-19.0.0-5812.g6f386a3f.el9.x86_64 536 kB/s | 125 kB 00:00 2024-08-16T02:35:40.930 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:40.930 INFO:teuthology.orchestra.run.smithi137.stdout:Total 532 kB/s | 125 kB 00:00 2024-08-16T02:35:40.931 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:40.934 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:40.934 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:40.997 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:40.998 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:41.149 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:41.150 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:41.150 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:41.150 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:41.150 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:41.150 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:41.155 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:41.258 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:41.461 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados-devel 2024-08-16T02:35:41.698 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:41.936 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:41.936 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:42.024 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:10 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:42.159 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:42.159 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:42.159 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 125 k 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 125 k 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 445 k 2024-08-16T02:35:42.160 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:42.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-08-16T02:35:42.382 INFO:teuthology.orchestra.run.smithi133.stdout:librados-devel-19.0.0-5812.g6f386a3f.el9.x86_64 555 kB/s | 125 kB 00:00 2024-08-16T02:35:42.383 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:42.383 INFO:teuthology.orchestra.run.smithi133.stdout:Total 552 kB/s | 125 kB 00:00 2024-08-16T02:35:42.383 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:42.388 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:42.389 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:42.452 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:42.453 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:42.610 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:42.706 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:42.774 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:08 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:42.866 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:42.911 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:42.912 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:35:42.912 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:42.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-08-16T02:35:43.155 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:43.416 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:43.416 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:43.416 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:43.416 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:43.416 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:43.416 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:43.526 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:09 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:43.657 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:43.658 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:43.658 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-08-16T02:35:43.658 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:43.658 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:43.658 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 32 k 2024-08-16T02:35:43.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:43.658 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:43.658 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:43.659 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-08-16T02:35:43.659 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:43.659 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 32 k 2024-08-16T02:35:43.659 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 142 k 2024-08-16T02:35:43.659 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:43.727 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs2 2024-08-16T02:35:43.834 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-19.0.0-5812.g6f386a3f.el9.x86_6 181 kB/s | 32 kB 00:00 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi137.stdout:Total 180 kB/s | 32 kB 00:00 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:43.838 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:43.838 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:43.853 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:43.853 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:43.937 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:44.039 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:12 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:44.373 INFO:teuthology.orchestra.run.smithi133.stdout:Package libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:44.416 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:44.417 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:35:44.417 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:44.438 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:44.482 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs-devel 2024-08-16T02:35:44.662 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:44.662 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:44.662 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:44.663 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:44.663 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:44.663 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:44.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-08-16T02:35:45.038 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:13 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:45.165 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:45.166 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:45.166 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-08-16T02:35:45.166 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:45.166 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:45.166 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 32 k 2024-08-16T02:35:45.166 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:45.166 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:45.166 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:45.167 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-08-16T02:35:45.167 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:45.167 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 32 k 2024-08-16T02:35:45.167 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 142 k 2024-08-16T02:35:45.167 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:45.341 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs-devel-19.0.0-5812.g6f386a3f.el9.x86_6 182 kB/s | 32 kB 00:00 2024-08-16T02:35:45.341 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:45.341 INFO:teuthology.orchestra.run.smithi133.stdout:Total 181 kB/s | 32 kB 00:00 2024-08-16T02:35:45.342 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:45.345 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:45.345 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:45.369 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:45.369 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:45.388 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:11 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:45.449 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:45.477 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:45.521 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:45.522 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:35:45.522 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:45.562 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:45.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-08-16T02:35:45.930 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:46.141 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:12 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:46.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:46.164 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:46.164 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:46.164 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:46.164 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:46.165 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:46.230 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:46.273 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:46.274 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:35:46.274 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:46.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-08-16T02:35:46.347 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados2 2024-08-16T02:35:46.896 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:12 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:46.902 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:14 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:46.991 INFO:teuthology.orchestra.run.smithi133.stdout:Package librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:47.034 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:47.036 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:35:47.037 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:47.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-08-16T02:35:47.121 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librbd1 2024-08-16T02:35:47.655 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:13 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:47.675 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:15 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:47.740 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:47.764 INFO:teuthology.orchestra.run.smithi133.stdout:Package librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:47.783 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:47.784 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:35:47.784 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:47.807 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:47.808 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:35:47.808 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:47.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-08-16T02:35:47.871 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rados 2024-08-16T02:35:48.411 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:14 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:48.433 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:16 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:48.497 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:48.518 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:48.540 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:48.541 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:35:48.541 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:48.561 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:48.562 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:35:48.562 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:48.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-08-16T02:35:48.626 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rgw 2024-08-16T02:35:49.165 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:15 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:49.185 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:17 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:49.250 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:49.271 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-08-16T02:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:49.313 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:49.314 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:35:49.314 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:49.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-08-16T02:35:49.376 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-cephfs 2024-08-16T02:35:49.916 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:15 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:49.923 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:17 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:50.008 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:50.044 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:50.044 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:50.044 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 91 k 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 91 k 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 321 k 2024-08-16T02:35:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:50.051 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:50.052 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:35:50.052 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:50.120 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rbd 2024-08-16T02:35:50.247 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 451 kB/s | 91 kB 00:00 2024-08-16T02:35:50.248 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:50.248 INFO:teuthology.orchestra.run.smithi137.stdout:Total 448 kB/s | 91 kB 00:00 2024-08-16T02:35:50.248 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:50.257 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:50.257 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:50.320 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:50.320 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:50.492 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:50.679 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:18 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:50.766 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:50.809 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:50.810 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-08-16T02:35:50.810 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:50.880 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-fuse 2024-08-16T02:35:50.983 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:51.263 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:51.263 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:51.263 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:51.263 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:51.264 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:51.264 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:51.434 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:19 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:51.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-08-16T02:35:51.566 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:51.566 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 91 k 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 91 k 2024-08-16T02:35:51.567 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 321 k 2024-08-16T02:35:51.568 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:51.794 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 402 kB/s | 91 kB 00:00 2024-08-16T02:35:51.795 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:51.795 INFO:teuthology.orchestra.run.smithi133.stdout:Total 400 kB/s | 91 kB 00:00 2024-08-16T02:35:51.795 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:51.804 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:51.804 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:51.867 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:51.867 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:52.020 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:52.075 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:18 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:52.120 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:52.205 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:52.205 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.5 M 2024-08-16T02:35:52.206 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 30 M 2024-08-16T02:35:52.207 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:52.489 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:52.571 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 9.6 MB/s | 3.5 MB 00:00 2024-08-16T02:35:52.572 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:52.572 INFO:teuthology.orchestra.run.smithi137.stdout:Total 9.5 MB/s | 3.5 MB 00:00 2024-08-16T02:35:52.572 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:52.583 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:52.583 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:52.665 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:52.665 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:52.744 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:52.744 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:52.744 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:52.744 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:52.744 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:52.744 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:53.016 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-mirror 2024-08-16T02:35:53.242 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:53.305 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:53.330 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:53.330 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:53.330 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-16T02:35:53.330 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:35:53.330 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:35:53.330 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:53.576 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:21 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:53.705 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:53.706 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:53.706 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-08-16T02:35:53.706 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:53.706 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:53.706 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:35:53.707 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:53.707 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:53.707 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:53.707 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-08-16T02:35:53.707 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:53.707 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 3.5 M 2024-08-16T02:35:53.707 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 30 M 2024-08-16T02:35:53.707 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:54.058 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-mirror-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 10 MB/s | 3.5 MB 00:00 2024-08-16T02:35:54.058 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:54.059 INFO:teuthology.orchestra.run.smithi133.stdout:Total 9.9 MB/s | 3.5 MB 00:00 2024-08-16T02:35:54.059 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:54.070 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:54.070 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:54.152 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:54.152 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:54.606 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:54.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:54.606 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:54.606 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:54.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:54.606 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:54.730 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:54.792 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:54.817 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:54.817 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:54.817 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-16T02:35:54.817 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:35:54.817 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:35:54.817 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:54.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-08-16T02:35:55.426 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:21 ago on Fri 16 Aug 2024 02:33:34 AM UTC. 2024-08-16T02:35:55.557 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-08-16T02:35:55.558 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:55.558 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-08-16T02:35:55.558 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:55.558 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 177 k 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 177 k 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 685 k 2024-08-16T02:35:55.559 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-08-16T02:35:55.783 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 789 kB/s | 177 kB 00:00 2024-08-16T02:35:55.784 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:55.785 INFO:teuthology.orchestra.run.smithi137.stdout:Total 783 kB/s | 177 kB 00:00 2024-08-16T02:35:55.785 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-08-16T02:35:55.794 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-08-16T02:35:55.795 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-08-16T02:35:55.858 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-08-16T02:35:55.859 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-08-16T02:35:56.021 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-08-16T02:35:56.123 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:56.124 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:56.124 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:56.124 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:56.124 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:56.124 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:56.131 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:56.377 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-nbd 2024-08-16T02:35:56.561 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:56.804 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:56.804 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:56.804 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-08-16T02:35:56.804 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:56.804 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:35:56.804 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-08-16T02:35:56.953 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:24 ago on Fri 16 Aug 2024 02:33:32 AM UTC. 2024-08-16T02:35:57.057 DEBUG:teuthology.parallel:result is None 2024-08-16T02:35:57.085 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 177 k 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-08-16T02:35:57.086 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-08-16T02:35:57.087 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:57.087 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 177 k 2024-08-16T02:35:57.087 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 685 k 2024-08-16T02:35:57.087 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-08-16T02:35:57.312 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-nbd-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 789 kB/s | 177 kB 00:00 2024-08-16T02:35:57.312 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:57.313 INFO:teuthology.orchestra.run.smithi133.stdout:Total 783 kB/s | 177 kB 00:00 2024-08-16T02:35:57.313 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-08-16T02:35:57.322 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-08-16T02:35:57.323 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-08-16T02:35:57.387 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-08-16T02:35:57.387 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-08-16T02:35:57.592 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-08-16T02:35:57.685 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:58.104 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:58.363 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:35:58.363 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:58.363 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-08-16T02:35:58.364 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:58.364 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:35:58.364 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-08-16T02:35:58.621 DEBUG:teuthology.parallel:result is None 2024-08-16T02:35:58.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:35:58.765 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-16T02:35:58.787 INFO:teuthology.orchestra.run.smithi133.stdout:19.0.0-5812.g6f386a3f.el9 2024-08-16T02:35:58.787 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5812.g6f386a3f.el9 2024-08-16T02:35:58.787 INFO:teuthology.task.install:The correct ceph version 19.0.0-5812.g6f386a3f is installed. 2024-08-16T02:35:58.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:35:58.931 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-16T02:35:58.953 INFO:teuthology.orchestra.run.smithi137.stdout:19.0.0-5812.g6f386a3f.el9 2024-08-16T02:35:58.954 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5812.g6f386a3f.el9 2024-08-16T02:35:58.954 INFO:teuthology.task.install:The correct ceph version 19.0.0-5812.g6f386a3f is installed. 2024-08-16T02:35:58.955 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-16T02:35:58.955 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:35:58.956 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-16T02:35:58.980 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:35:58.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-16T02:35:59.019 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-16T02:35:59.019 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:35:59.019 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2024-08-16T02:35:59.045 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-16T02:35:59.110 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:35:59.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-08-16T02:35:59.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-16T02:35:59.199 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-16T02:35:59.199 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:35:59.199 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-16T02:35:59.224 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-16T02:35:59.288 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:35:59.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-16T02:35:59.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-16T02:35:59.377 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-16T02:35:59.377 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:35:59.377 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/stdin-killer 2024-08-16T02:35:59.402 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-16T02:35:59.467 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:35:59.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-08-16T02:35:59.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-16T02:35:59.556 INFO:teuthology.run_tasks:Running task ceph... 2024-08-16T02:35:59.637 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-16T02:35:59.638 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /var/log/ceph 2024-08-16T02:35:59.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-08-16T02:35:59.664 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-16T02:35:59.664 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-16T02:35:59.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-16T02:35:59.729 INFO:tasks.ceph:Creating extra log directories... 2024-08-16T02:35:59.729 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-16T02:35:59.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-16T02:35:59.803 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-16T02:35:59.803 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-16T02:35:59.803 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420', 'branch': 'wip-yingxin-testing-2024-08-15-1551', 'description': 'crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7857420', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-testing-2024-08-15-1551'}, 'ceph': {'conf': {'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}}, 'workunit': {'branch': 'wip-yingxin-testing-2024-08-15-1551', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-yingxin-testing-2024-08-15-1551', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'targets': {'smithi133.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBkKGkigL0azSs8Ojv+rogQ7hOn1HRKC6g2yWXMDId1EIKYb03B3s0YUmeL9qbrrcUEgEqgYc3hRLDJRjA2HIjs=', 'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsJzJKq6v8ELjQkdDZGhIL3RYLFtefFQhp2GXslHlmUNW3mbUqOMgrqHE/i167yzNoINffqqQIuALeW9I/lqOs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rados/test.sh --crimson', 'rados/test_pool_quota.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '67db276914d29505c9707d04f3a3b8fb9d8aee4b', 'timestamp': '2024-08-16_01:59:29', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-16T02:35:59.804 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-16T02:35:59.835 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-16T02:35:59.851 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2024-08-16T02:35:59.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-08-16T02:35:59.916 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:35:59.916 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2024-08-16T02:35:59.969 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-16T02:35:59.970 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 2024-08-16T02:36:00.030 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-16T02:36:00.030 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:00.030 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-08-16T02:36:00.030 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:00.030 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:00.031 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-08-16 02:35:55.075608150 +0000 2024-08-16T02:36:00.031 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-08-16 02:32:33.285887864 +0000 2024-08-16T02:36:00.031 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-08-16 02:32:33.285887864 +0000 2024-08-16T02:36:00.031 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-08-16 02:32:33.285887864 +0000 2024-08-16T02:36:00.031 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-16T02:36:00.093 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-08-16T02:36:00.093 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-08-16T02:36:00.093 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000207627 s, 2.5 MB/s 2024-08-16T02:36:00.095 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-16T02:36:00.151 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 2024-08-16T02:36:00.205 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-16T02:36:00.206 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:00.206 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-08-16T02:36:00.206 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:00.206 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:00.206 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-08-16 02:35:55.075608150 +0000 2024-08-16T02:36:00.206 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-08-16 02:32:33.622881122 +0000 2024-08-16T02:36:00.206 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-08-16 02:32:33.622881122 +0000 2024-08-16T02:36:00.206 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-08-16 02:32:33.622881122 +0000 2024-08-16T02:36:00.206 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-16T02:36:00.268 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-08-16T02:36:00.268 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-08-16T02:36:00.269 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000181997 s, 2.8 MB/s 2024-08-16T02:36:00.270 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-16T02:36:00.326 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 2024-08-16T02:36:00.382 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-16T02:36:00.383 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:00.383 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-16T02:36:00.383 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:00.383 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:00.383 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-08-16 02:35:55.075608150 +0000 2024-08-16T02:36:00.383 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-08-16 02:32:33.954874478 +0000 2024-08-16T02:36:00.383 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-08-16 02:32:33.954874478 +0000 2024-08-16T02:36:00.383 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-08-16 02:32:33.954874478 +0000 2024-08-16T02:36:00.383 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-16T02:36:00.445 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-08-16T02:36:00.446 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-08-16T02:36:00.446 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000210836 s, 2.4 MB/s 2024-08-16T02:36:00.447 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-16T02:36:00.504 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 2024-08-16T02:36:00.559 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-16T02:36:00.559 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:00.559 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-08-16T02:36:00.559 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:00.560 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:00.560 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-08-16 02:35:55.076608130 +0000 2024-08-16T02:36:00.560 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-08-16 02:32:34.277868014 +0000 2024-08-16T02:36:00.560 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-08-16 02:32:34.277868014 +0000 2024-08-16T02:36:00.560 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-08-16 02:32:34.277868014 +0000 2024-08-16T02:36:00.560 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-16T02:36:00.623 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-08-16T02:36:00.623 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-08-16T02:36:00.623 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000216955 s, 2.4 MB/s 2024-08-16T02:36:00.624 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-16T02:36:00.681 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:00.681 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:00.681 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-08-16T02:36:00.701 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-16T02:36:00.701 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-08-16T02:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-16T02:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-08-16T02:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-08-16 02:35:53.593662347 +0000 2024-08-16T02:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-08-16 02:32:33.754069579 +0000 2024-08-16T02:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-08-16 02:32:33.754069579 +0000 2024-08-16T02:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-08-16 02:32:33.754069579 +0000 2024-08-16T02:36:00.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-16T02:36:00.825 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-08-16T02:36:00.825 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-08-16T02:36:00.825 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000303067 s, 1.7 MB/s 2024-08-16T02:36:00.826 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-16T02:36:00.885 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-08-16T02:36:00.942 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-16T02:36:00.943 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:00.943 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-08-16T02:36:00.943 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:00.943 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:00.943 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-08-16 02:35:53.593662347 +0000 2024-08-16T02:36:00.943 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-08-16 02:32:34.101062864 +0000 2024-08-16T02:36:00.943 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-08-16 02:32:34.101062864 +0000 2024-08-16T02:36:00.943 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-08-16 02:32:34.101062864 +0000 2024-08-16T02:36:00.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-16T02:36:01.008 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-08-16T02:36:01.008 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-08-16T02:36:01.008 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.00022753 s, 2.3 MB/s 2024-08-16T02:36:01.009 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-16T02:36:01.067 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-08-16 02:35:53.594662329 +0000 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-08-16 02:32:34.440056304 +0000 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-08-16 02:32:34.440056304 +0000 2024-08-16T02:36:01.124 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-08-16 02:32:34.440056304 +0000 2024-08-16T02:36:01.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-16T02:36:01.188 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-08-16T02:36:01.188 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-08-16T02:36:01.188 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.00023462 s, 2.2 MB/s 2024-08-16T02:36:01.190 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-16T02:36:01.247 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-08-16 02:35:53.594662329 +0000 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-08-16 02:32:34.768049957 +0000 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-08-16 02:32:34.768049957 +0000 2024-08-16T02:36:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-08-16 02:32:34.768049957 +0000 2024-08-16T02:36:01.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-16T02:36:01.368 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-08-16T02:36:01.368 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-08-16T02:36:01.368 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000193192 s, 2.7 MB/s 2024-08-16T02:36:01.369 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-16T02:36:01.426 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:01.426 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-08-16T02:36:01.426 INFO:tasks.ceph:Generating config... 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-16T02:36:01.428 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-08-16T02:36:01.429 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] debug monc = 20 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] osd class default list = * 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] osd class load list = * 2024-08-16T02:36:01.429 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-08-16T02:36:01.430 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-08-16T02:36:01.430 INFO:tasks.ceph:[client] debug ms = 1 2024-08-16T02:36:01.430 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:36:01.430 INFO:tasks.ceph:[global] ms client mode = crc 2024-08-16T02:36:01.430 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-08-16T02:36:01.430 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-08-16T02:36:01.430 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-08-16T02:36:01.430 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-08-16T02:36:01.430 INFO:tasks.ceph:[global] ms service mode = crc 2024-08-16T02:36:01.430 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-16T02:36:01.430 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-16T02:36:01.430 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-08-16T02:36:01.430 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-08-16T02:36:01.431 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-08-16T02:36:01.431 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-08-16T02:36:01.431 INFO:tasks.ceph:Setting up mon.a... 2024-08-16T02:36:01.431 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-16T02:36:01.495 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/ceph.keyring 2024-08-16T02:36:01.498 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-16T02:36:01.551 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:36:01.575 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.133')] 2024-08-16T02:36:01.576 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.133', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug ms': 1}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-08-16T02:36:01.577 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:01.577 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-16T02:36:01.631 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.133 --print /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:01.722+0000 7f864d02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:01.722+0000 7f864d02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:01.722+0000 7f864d02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: generated fsid 7a5c695b-6aa6-4555-8e56-4a247af82439 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:setting min_mon_release = quincy 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:epoch 0 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:fsid 7a5c695b-6aa6-4555-8e56-4a247af82439 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:last_changed 2024-08-16T02:36:01.723741+0000 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:created 2024-08-16T02:36:01.723741+0000 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:min_mon_release 17 (quincy) 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:election_strategy: 1 2024-08-16T02:36:01.724 INFO:teuthology.orchestra.run.smithi133.stdout:0: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.a 2024-08-16T02:36:01.725 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-08-16T02:36:01.726 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-16T02:36:01.740 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7a5c695b-6aa6-4555-8e56-4a247af82439... 2024-08-16T02:36:01.741 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-16T02:36:01.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.823 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2024-08-16T02:36:01.824 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: mon_allow_pool_size_one = true 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.825 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: mon health detail to clog = false 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = 172.21.15.133 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: ms client mode = crc 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: ms cluster mode = crc 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: ms mon client mode = crc 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: ms mon cluster mode = crc 2024-08-16T02:36:01.826 INFO:teuthology.orchestra.run.smithi133.stdout: ms mon service mode = crc 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: ms service mode = crc 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = 7a5c695b-6aa6-4555-8e56-4a247af82439 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: osd mclock profile = high_recovery_ops 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.827 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug trim objects = true 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2024-08-16T02:36:01.828 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: debug monc = 20 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: crimson alien thread cpu cores = 6-7 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: crimson osd obc lru size = 10 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 20 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: osd blocked scrub grace period = 3600 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: osd class default list = * 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: osd class load list = * 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: osd objectstore = seastore 2024-08-16T02:36:01.829 INFO:teuthology.orchestra.run.smithi133.stdout: osd shutdown pgref assert = True 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2024-08-16T02:36:01.830 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: auth mon ticket ttl = 660 # 11m 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: auth service ticket ttl = 240 # 4m 2024-08-16T02:36:01.831 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: # don't complain about insecure global_id in the test suite 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: # 1m isn't quite enough 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon_down_mkfs_grace = 2m 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_filestore_osds = false 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon min osdmap epochs = 50 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune interval = 2 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune min = 15 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune txsize = 2 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: paxos service trim min = 10 2024-08-16T02:36:01.832 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on pool no app = False 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default crimson = True 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: crimson seastar cpu cores = 0-2 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout: crimson seastar cpu cores = 3-5 2024-08-16T02:36:01.833 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2024-08-16T02:36:01.834 INFO:teuthology.orchestra.run.smithi133.stdout: crimson seastar cpu cores = 0-2 2024-08-16T02:36:01.834 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.3] 2024-08-16T02:36:01.834 INFO:teuthology.orchestra.run.smithi133.stdout: crimson seastar cpu cores = 3-5 2024-08-16T02:36:01.834 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-08-16T02:36:01.834 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.835 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-08-16T02:36:01.836 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-08-16T02:36:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.133 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: ms client mode = crc 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: ms cluster mode = crc 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: ms mon client mode = crc 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: ms mon cluster mode = crc 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: ms mon service mode = crc 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: ms service mode = crc 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 7a5c695b-6aa6-4555-8e56-4a247af82439 2024-08-16T02:36:01.838 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-08-16T02:36:01.839 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug trim objects = true 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-08-16T02:36:01.840 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: debug monc = 20 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: crimson alien thread cpu cores = 6-7 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: crimson osd obc lru size = 10 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: osd blocked scrub grace period = 3600 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: osd class default list = * 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: osd class load list = * 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = seastore 2024-08-16T02:36:01.841 INFO:teuthology.orchestra.run.smithi137.stdout: osd shutdown pgref assert = True 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-08-16T02:36:01.842 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.843 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon min osdmap epochs = 50 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon osdmap full prune interval = 2 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon osdmap full prune min = 15 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon osdmap full prune txsize = 2 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: paxos service trim min = 10 2024-08-16T02:36:01.844 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on pool no app = False 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default crimson = True 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-08-16T02:36:01.845 INFO:teuthology.orchestra.run.smithi137.stdout: crimson seastar cpu cores = 0-2 2024-08-16T02:36:01.846 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-08-16T02:36:01.846 INFO:teuthology.orchestra.run.smithi137.stdout: crimson seastar cpu cores = 3-5 2024-08-16T02:36:01.846 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-08-16T02:36:01.846 INFO:teuthology.orchestra.run.smithi137.stdout: crimson seastar cpu cores = 0-2 2024-08-16T02:36:01.846 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.3] 2024-08-16T02:36:01.846 INFO:teuthology.orchestra.run.smithi137.stdout: crimson seastar cpu cores = 3-5 2024-08-16T02:36:01.847 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-16T02:36:01.848 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-16T02:36:01.951 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-16T02:36:01.951 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:01.951 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-16T02:36:01.966 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:01.967 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-16T02:36:02.022 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2024-08-16T02:36:02.022 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:02.022 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-16T02:36:02.022 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:36:02.096 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:02.096 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:02.151 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-08-16T02:36:02.151 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:02.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-16T02:36:02.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:36:02.188 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:02.188 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:02.246 INFO:tasks.ceph:Setting up mon nodes... 2024-08-16T02:36:02.246 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-16T02:36:02.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-16T02:36:02.353 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-16T02:36:02.357 INFO:tasks.ceph:Setting up mds nodes... 2024-08-16T02:36:02.357 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-16T02:36:02.357 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-16T02:36:02.405 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-16T02:36:02.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-16T02:36:02.468 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-08-16T02:36:02.481 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-16T02:36:02.481 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-08-16T02:36:02.482 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-16T02:36:02.506 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:02.506 INFO:tasks.ceph:role: osd.0 2024-08-16T02:36:02.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com 2024-08-16T02:36:02.506 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-16T02:36:02.578 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:36:02.579 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:36:03.093 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2024-08-16T02:36:03.095 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2024-08-16T02:36:03.095 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-16T02:36:03.238 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-16T02:36:03.270 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-16T02:36:03.336 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:03.336 INFO:tasks.ceph:role: osd.1 2024-08-16T02:36:03.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com 2024-08-16T02:36:03.337 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:36:03.410 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:36:03.853 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2024-08-16T02:36:03.856 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2024-08-16T02:36:03.856 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-16T02:36:03.939 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-16T02:36:04.009 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:04.324 INFO:teuthology.orchestra.run.smithi133.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:04.325 INFO:teuthology.orchestra.run.smithi133.stderr:INFO 2024-08-16 02:36:04,326 seastar - Reactor backend: linux-aio 2024-08-16T02:36:04.337 INFO:teuthology.orchestra.run.smithi133.stderr:==30021==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:04.447 INFO:teuthology.orchestra.run.smithi133.stderr:INFO 2024-08-16 02:36:04,447 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:36:04.447 INFO:teuthology.orchestra.run.smithi133.stderr:INFO 2024-08-16 02:36:04,447 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-08-16T02:36:04.447 INFO:teuthology.orchestra.run.smithi133.stderr:DEBUG 2024-08-16 02:36:04,448 [shard 0:main] seastar - reactor::drain 2024-08-16T02:36:04.988 INFO:teuthology.orchestra.run.smithi133.stderr:----------------------------------------------------- 2024-08-16T02:36:04.988 INFO:teuthology.orchestra.run.smithi133.stderr:Suppressions used: 2024-08-16T02:36:04.988 INFO:teuthology.orchestra.run.smithi133.stderr: count bytes template 2024-08-16T02:36:04.988 INFO:teuthology.orchestra.run.smithi133.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:04.988 INFO:teuthology.orchestra.run.smithi133.stderr:----------------------------------------------------- 2024-08-16T02:36:04.989 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-08-16T02:36:05.049 INFO:teuthology.orchestra.run.smithi133.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:05.053 INFO:teuthology.orchestra.run.smithi133.stderr:INFO 2024-08-16 02:36:05,053 seastar - Reactor backend: linux-aio 2024-08-16T02:36:05.061 INFO:teuthology.orchestra.run.smithi133.stderr:==30019==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:05.327 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xdfd39 0xb93070d 0xb9307c0 0xbb030a1 0xbb03190 0xbb03410 0xbb034c9 0xbb039bb 0xbb03e7d 0xbaa8db3 0xbb04814 0xbb04a5b 0xbb04b0b 0xbab55cb 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.327 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x95a3edd 0x981ce23 0x9ae623a 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.327 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.327 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.473 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36297 0x5c2b16e 0xbac779d 0xbac7a04 0xbac97bf 0xbac9c0a 0xbab5fba 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x923b1fc 0x923b718 0x923b8a1 0x923bae8 0x923bdc0 0x923be75 0x9556f6e 0x95571d3 0x958fe43 0x96a9a42 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.473 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.498 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x9e4b348 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:05.498 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.556 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8a39 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:05.556 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.588 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bcd 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.588 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.588 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8393 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.588 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.645 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7eef 0x9e4b36e 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.646 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.646 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x350349f 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.646 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.663 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bcd 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:05.663 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.927 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x105bc0 0x2a517 0xddd2d 0x36e7625 0x5c5759d 0x5c57903 0x5c579f8 0xbaf968a 0xbb04840 0xbb04a5b 0xbb04b0b 0xbab55cb 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.928 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113cb0 0x4a0bd 0x35b7c 0x3629e 0xbac61ab 0xbac75e2 0xbac785c 0xbac7a04 0xbac97bf 0xbac99bb 0xbb0dc39 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:05.928 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:05.928 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:06.010 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d88e8 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:06.010 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:06.068 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7eef 0x9e4b36e 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:06.068 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:06.073 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x35b6f 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:06.074 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:06.939 INFO:teuthology.orchestra.run.smithi133.stderr:----------------------------------------------------- 2024-08-16T02:36:06.939 INFO:teuthology.orchestra.run.smithi133.stderr:Suppressions used: 2024-08-16T02:36:06.939 INFO:teuthology.orchestra.run.smithi133.stderr: count bytes template 2024-08-16T02:36:06.939 INFO:teuthology.orchestra.run.smithi133.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:06.939 INFO:teuthology.orchestra.run.smithi133.stderr:----------------------------------------------------- 2024-08-16T02:36:06.939 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-08-16T02:36:06.968 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-16T02:36:06.992 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:07.291 INFO:teuthology.orchestra.run.smithi133.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:07.292 INFO:teuthology.orchestra.run.smithi133.stderr:INFO 2024-08-16 02:36:07,292 seastar - Reactor backend: linux-aio 2024-08-16T02:36:07.297 INFO:teuthology.orchestra.run.smithi133.stderr:==30068==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:07.408 INFO:teuthology.orchestra.run.smithi133.stderr:INFO 2024-08-16 02:36:07,408 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:36:07.408 INFO:teuthology.orchestra.run.smithi133.stderr:INFO 2024-08-16 02:36:07,408 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-08-16T02:36:07.408 INFO:teuthology.orchestra.run.smithi133.stderr:DEBUG 2024-08-16 02:36:07,408 [shard 0:main] seastar - reactor::drain 2024-08-16T02:36:07.949 INFO:teuthology.orchestra.run.smithi133.stderr:----------------------------------------------------- 2024-08-16T02:36:07.949 INFO:teuthology.orchestra.run.smithi133.stderr:Suppressions used: 2024-08-16T02:36:07.949 INFO:teuthology.orchestra.run.smithi133.stderr: count bytes template 2024-08-16T02:36:07.949 INFO:teuthology.orchestra.run.smithi133.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:07.949 INFO:teuthology.orchestra.run.smithi133.stderr:----------------------------------------------------- 2024-08-16T02:36:07.949 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-08-16T02:36:08.009 INFO:teuthology.orchestra.run.smithi133.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:08.013 INFO:teuthology.orchestra.run.smithi133.stderr:INFO 2024-08-16 02:36:08,013 seastar - Reactor backend: linux-aio 2024-08-16T02:36:08.022 INFO:teuthology.orchestra.run.smithi133.stderr:==30066==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:08.087 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113c62 0x4a0bd 0x35b7c 0x36eee 0x5f73f72 0x5f74045 0x5fcf264 0x5f4f446 0x5f4f97a 0x5d929a1 0x5daabe3 0x70ee632 0x70ffe05 0x71012a2 0x7101327 0x7101429 0x7101521 0x710172e 0x7101914 0x7101c63 0x7102007 0x7102237 0x71024a1 0x710276e 0x710298c 0x7102a44 0x7102b0a 0x7102beb 0x5bfe1b8 0x5bfe2c2 0x5bfe37e 0x5bfe43a 0x5bfe4f6 0x5bfe643 0x5bfe7ce 0x5bfeb8f 0x5c1aad0 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:08.087 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.252 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x4a0ae 0x35b7c 0x3629e 0x5c2b16e 0xbac784a 0xbac7a04 0xbac97bf 0xbac9c0a 0xbab5fba 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x967d442 0x967d7de 0x967d963 0x967da05 0x967db1e 0xbf2c63b 0xbf2c842 0xbf2c918 0xbf2ca53 0xbf2ce47 0xbf2c20a 0xbf2c2ea 0xbf2c3fe 0xbf2c0c1 0x94c7872 0x95add6e 0x92ea331 0x92ea50b 0x92ea646 0x92ea700 0x92ea7da 0x92ead26 0x92eaffe 0x92eb0b3 0x9561672 0x95618d7 0x958fd08 0x96a8aca 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:08.253 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.302 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x1068e3 0xec679 0xe0ec6 0x5c57472 0x5c5764c 0x5c57903 0x5c579f8 0x95fa118 0xbaf914a 0xbaf9366 0xbaa79e0 0x9ae6343 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.302 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.302 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x3afc7d4 0x3afc9ba 0xbac630f 0xbac75e2 0xbac785c 0xbac7a04 0xbb0e0ed 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.302 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.360 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x105f18 0xec679 0xdf267 0x37ef979 0xbab869a 0xbaa4bc4 0xbaa4fa8 0x9e29a36 0x9e39d7e 0x99e55e7 0x99e5bed 0x98decca 0x92f408f 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.360 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.360 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bdf 0x4a0bd 0x35b7c 0x36eee 0x9efa688 0x9efa7c0 0x9efaa0a 0x9efb136 0x9efb410 0x9efb7b8 0x9efeb92 0x9eff7ee 0x9effab1 0x9f00728 0x9e39732 0x99e55e7 0x99e5bed 0x98decca 0x92f408f 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.360 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.499 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x9e4b334 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:08.499 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.557 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bed 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:08.557 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.588 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x3631b 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.589 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.646 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bcd 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.646 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.664 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x3631b 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:08.664 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.929 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x105eb0 0xec679 0xdf267 0x37ef979 0xb9302f2 0xb93043a 0xb930523 0xb9307e3 0xb9308d6 0xb9309b6 0xb930c36 0xb9311e1 0xb9315ec 0xb9318ab 0xbb027c5 0xbb0601c 0xbab9ea9 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.929 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x2d587 0x2d86b 0x29edb 0xdfdb3 0xb93070d 0xb9307c0 0xb9308d6 0xb9309b6 0xb930c36 0xb930cef 0xb9311e1 0xb9315ec 0xb9318ab 0xb932368 0xbb021f2 0xbaa819b 0x9ae5c91 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.929 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.929 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:08.987 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xfada3 0xfef5d 0x2cd58 0x2d2fe 0x2d3ae 0x2d688 0x2d86b 0x29edb 0xdfdb3 0xbb04a3b 0xbb04b0b 0xbab55cb 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9e29a36 0x9e39d7e 0x99e55e7 0x99e5bed 0x98dff16 0x92d84df 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:08.987 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff98088c2c 0xffffffff9808fb6e 0xffffffff980904e4 0xffffffff97d9a865 0xffffffff97d9e6eb 0xffffffff97d9ed7d 0xffffffff97d6b64d 0xffffffff986c445f 0xffffffff9880012b 2024-08-16T02:36:09.078 INFO:teuthology.orchestra.run.smithi133.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:09.079 INFO:teuthology.orchestra.run.smithi133.stderr:kernel callstack: 2024-08-16T02:36:09.968 INFO:teuthology.orchestra.run.smithi133.stderr:----------------------------------------------------- 2024-08-16T02:36:09.968 INFO:teuthology.orchestra.run.smithi133.stderr:Suppressions used: 2024-08-16T02:36:09.969 INFO:teuthology.orchestra.run.smithi133.stderr: count bytes template 2024-08-16T02:36:09.969 INFO:teuthology.orchestra.run.smithi133.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:09.969 INFO:teuthology.orchestra.run.smithi133.stderr:----------------------------------------------------- 2024-08-16T02:36:09.969 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-08-16T02:36:09.996 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-16T02:36:10.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-16T02:36:10.052 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:10.052 INFO:tasks.ceph:role: osd.2 2024-08-16T02:36:10.052 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-08-16T02:36:10.053 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-16T02:36:10.129 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:36:10.129 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:36:10.130 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:36:10.130 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:36:10.130 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:36:10.130 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:36:10.130 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:36:10.130 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:36:10.130 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:36:10.130 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:36:10.635 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-08-16T02:36:10.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-08-16T02:36:10.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-08-16T02:36:10.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-16T02:36:10.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-16T02:36:10.885 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:10.885 INFO:tasks.ceph:role: osd.3 2024-08-16T02:36:10.885 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-08-16T02:36:10.885 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:36:10.960 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:36:11.403 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-08-16T02:36:11.404 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-08-16T02:36:11.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-08-16T02:36:11.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-16T02:36:11.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:11.871 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:11.872 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-08-16 02:36:11,873 seastar - Reactor backend: linux-aio 2024-08-16T02:36:11.890 INFO:teuthology.orchestra.run.smithi137.stderr:==29251==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:11.995 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x155f3 0x4a09d 0x35b7c 0x3689e 0x379f42c 0x5f73807 0x5f73f1f 0x5f74045 0x5fcf264 0x5f4f446 0x5f4f620 0x5f4fbac 0x5d929a1 0x5daabe3 0x70ee632 0x70ffe05 0x71012a2 0x7101327 0x7101429 0x7101521 0x710172e 0x7101914 0x7101c63 0x7102007 0x7102237 0x71024a1 0x710276e 0x710298c 0x7102a44 0x7102b0a 0x7102beb 0x5bfe1b8 0x5bfe2c2 0x5bfe37e 0x5bfe43a 0x5bfe4f6 0x5bfe643 0x5bfe7ce 0x5bfeb8f 0x5c1aad0 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x38e8d78 0x38e98d1 0x3788ab2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:11.995 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:12.006 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-08-16 02:36:12,006 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:36:12.006 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-08-16 02:36:12,006 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-08-16T02:36:12.006 INFO:teuthology.orchestra.run.smithi137.stderr:DEBUG 2024-08-16 02:36:12,007 [shard 0:main] seastar - reactor::drain 2024-08-16T02:36:12.583 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-08-16T02:36:12.583 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-08-16T02:36:12.583 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-08-16T02:36:12.583 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:12.584 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-08-16T02:36:12.584 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-08-16T02:36:12.639 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:12.663 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-08-16 02:36:12,643 seastar - Reactor backend: linux-aio 2024-08-16T02:36:12.663 INFO:teuthology.orchestra.run.smithi137.stderr:==29247==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:12.912 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xbb03af7 0xbb03e7d 0xbb042d5 0xbaa8c90 0xbb04814 0xbb04a5b 0xbb04b0b 0xbab55cb 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:12.912 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:12.912 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x25e3f 0x35b7c 0x3689e 0x3afc839 0x3afc9ba 0xbacd073 0xbacd328 0xbab5b7f 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:12.912 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:12.945 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x8b51c 0xef7ec 0x2d848 0x29edb 0xdfdb3 0xb93070d 0xb9307c0 0xbb030a1 0xbb03190 0xbb03850 0xbb039bb 0xbb03e7d 0xbb042d5 0xbaa8c90 0xbb04814 0xbb04a5b 0xbb04b0b 0xbab55cb 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x923b1fc 0x923b718 0x923b8a1 0x923bae8 0x923bdc0 0x923be75 0x9556f6e 0x95571d3 0x958fe43 0x96a9a42 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:12.945 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.003 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36f52ba 0x3afc7b4 0x3afc9ba 0xbacd073 0xbacd328 0xbab5b7f 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x923b1fc 0x923b718 0x923b8a1 0x923bae8 0x923bdc0 0x923be75 0x9556f6e 0x95571d3 0x958fe43 0x96a9a42 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:13.003 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.083 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7a1c 0x54d89ce 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:13.083 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.141 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362fa 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:13.141 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.170 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x4a0ae 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:13.170 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.170 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d82e3 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:13.170 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.228 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8358 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:13.228 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.228 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x4a05f 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:13.228 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.248 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362fa 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:13.248 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.519 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x105bc0 0x2d93f 0x29c4a 0xdf1eb 0x37ef979 0xbaadb6e 0x981cebf 0x9ae5c54 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:13.519 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xbac797d 0xbac7a04 0xbac97bf 0xbac99bb 0xbb0dc39 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:13.519 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.519 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.665 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36289 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:13.665 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:13.666 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36331 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:13.666 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:14.542 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-08-16T02:36:14.542 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-08-16T02:36:14.542 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-08-16T02:36:14.542 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:14.542 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-08-16T02:36:14.543 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-08-16T02:36:14.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-16T02:36:14.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:14.898 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:14.899 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-08-16 02:36:14,900 seastar - Reactor backend: linux-aio 2024-08-16T02:36:14.904 INFO:teuthology.orchestra.run.smithi137.stderr:==29300==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:15.016 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-08-16 02:36:15,015 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:36:15.016 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-08-16 02:36:15,015 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-08-16T02:36:15.016 INFO:teuthology.orchestra.run.smithi137.stderr:DEBUG 2024-08-16 02:36:15,016 [shard 0:main] seastar - reactor::drain 2024-08-16T02:36:15.550 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-08-16T02:36:15.550 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-08-16T02:36:15.551 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-08-16T02:36:15.551 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:15.551 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-08-16T02:36:15.551 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-08-16T02:36:15.612 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:15.618 INFO:teuthology.orchestra.run.smithi137.stderr:INFO 2024-08-16 02:36:15,616 seastar - Reactor backend: linux-aio 2024-08-16T02:36:15.625 INFO:teuthology.orchestra.run.smithi137.stderr:==29298==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:15.892 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xc9b6f 0xca7a6 0x35920f0 0x3afc7c2 0x3afc9ba 0xbac630f 0xbac75e2 0xbac785c 0xbac7a04 0xbb0e0ed 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:15.892 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:15.892 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x5c2862e 0xbac7580 0xbac785c 0xbac7a04 0xbb0e0ed 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:15.892 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.065 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113ca1 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:16.065 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.123 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d838c 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:16.123 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.168 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bc7 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:16.168 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.168 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7a0a 0x54d89ce 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:16.168 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.225 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362f6 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:16.226 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.226 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d838c 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:16.226 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.230 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x8b523 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:16.230 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.507 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xef7f8 0x4a0a5 0x35b7c 0x3689e 0x3afc839 0x3afc9ba 0xbacd073 0xbad0ae1 0xbb13898 0xbab5c10 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:16.507 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36f5447 0x3afc716 0x3afc9ba 0xbacd073 0xbacd328 0xbab5b7f 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:16.507 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.507 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.589 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d82e3 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:16.589 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:16.647 INFO:teuthology.orchestra.run.smithi137.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d79e3 0x54d89ce 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:16.647 INFO:teuthology.orchestra.run.smithi137.stderr:kernel callstack: 2024-08-16T02:36:17.527 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-08-16T02:36:17.527 INFO:teuthology.orchestra.run.smithi137.stderr:Suppressions used: 2024-08-16T02:36:17.527 INFO:teuthology.orchestra.run.smithi137.stderr: count bytes template 2024-08-16T02:36:17.527 INFO:teuthology.orchestra.run.smithi137.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:17.527 INFO:teuthology.orchestra.run.smithi137.stderr:----------------------------------------------------- 2024-08-16T02:36:17.527 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-08-16T02:36:17.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-16T02:36:17.578 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-16T02:36:17.579 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:17.579 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-16T02:36:17.609 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:17.609 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-16T02:36:17.679 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:17.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-16T02:36:17.704 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:17.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-16T02:36:17.769 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:17.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-16T02:36:17.834 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:17.835 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-16T02:36:17.853 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:17.854 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-08-16T02:36:17.890 INFO:tasks.ceph:Adding keys to all mons... 2024-08-16T02:36:17.890 DEBUG:teuthology.orchestra.run.smithi133:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-16T02:36:17.921 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2024-08-16T02:36:17.921 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAVu75mHmCvCxAAsNGW3RGY7MfYK4Yu58PcmQ== 2024-08-16T02:36:17.921 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2024-08-16T02:36:17.921 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAYu75ms8yvCRAAe6LVu+R3EWBOM8cccCneZA== 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.x] 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQASu75m4dIaFRAAWmoJZh3CAdhs+3jFgUSwvw== 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAcu75m4ysMLxAA1rwG5gaafmAFYKs4FXfmFw== 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.3] 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAfu75mK4dILRAAQcZ4V8x/VukJxzf4kIvDfg== 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQASu75mpkA1GBAAxJ1yN9/qFPa57sK3j9hEAA== 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout:[client.1] 2024-08-16T02:36:17.922 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQASu75m0pXvGxAAqcOZrG98xOlg8wvIXwjexg== 2024-08-16T02:36:17.923 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:36:18.030 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:36:18.092 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-16T02:36:18.152 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:36:18.212 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:36:18.272 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-16T02:36:18.336 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-16T02:36:18.399 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-16T02:36:18.400 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-16T02:36:18.424 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-16T02:36:18.533 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.532+0000 7fb7538cac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.533 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.532+0000 7fb7538cac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.546 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-16T02:36:18.571 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:18.626 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-16T02:36:18.627 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-16T02:36:18.627 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-16T02:36:18.669 INFO:tasks.ceph.mon.a:Started 2024-08-16T02:36:18.669 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-16T02:36:18.670 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-16T02:36:18.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-16T02:36:18.672 INFO:tasks.ceph.mgr.x:Started 2024-08-16T02:36:18.672 DEBUG:tasks.ceph:set 0 configs 2024-08-16T02:36:18.673 INFO:tasks.ceph:Setting crush tunables to default 2024-08-16T02:36:18.673 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd crush tunables default 2024-08-16T02:36:18.751 INFO:tasks.ceph.mon.a.smithi133.stderr:2024-08-16T02:36:18.751+0000 7fbd18e0dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.752 INFO:tasks.ceph.mon.a.smithi133.stderr:2024-08-16T02:36:18.751+0000 7fbd18e0dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.752 INFO:tasks.ceph.mon.a.smithi133.stderr:2024-08-16T02:36:18.751+0000 7fbd18e0dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.765 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-08-16T02:36:18.764+0000 7fbad01c5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.782 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.775+0000 7fe34d82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.782+0000 7fe34d82d640 1 Processor -- start 2024-08-16T02:36:18.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.783+0000 7fe34d82d640 1 -- start start 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe34d82d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 0x7fe3481385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe34d82d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe348073460 con 0x7fe348071bd0 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe34d82d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7fe348073690 con 0x7fe3480726f0 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe3467fc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 0x7fe3481385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe3467fc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 0x7fe3481385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:39708/0 (socket says 172.21.15.133:39708) 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe3467fc640 1 -- 172.21.15.133:0/3356685109 learned_addr learned my addr 172.21.15.133:0/3356685109 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe3467fc640 1 -- 172.21.15.133:0/3356685109 >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 msgr2=0x7fe3481385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe3467fc640 1 -- 172.21.15.133:0/3356685109 >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 msgr2=0x7fe3481385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe3467fc640 1 --2- 172.21.15.133:0/3356685109 >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 0x7fe3481385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-16T02:36:18.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.784+0000 7fe3467fc640 1 --2- 172.21.15.133:0/3356685109 >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 0x7fe3481385b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-16T02:36:18.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 837203529 0 0) 0x7fe348073460 con 0x7fe348071bd0 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe330003580 con 0x7fe348071bd0 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 <== mon.0 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3396848151 0 0) 0x7fe330003580 con 0x7fe348071bd0 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 msgr2=0x7fe3481385b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 --2- 172.21.15.133:0/3356685109 >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 0x7fe3481385b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3480749a0 con 0x7fe348071bd0 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 <== mon.0 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 ==== 205+0+0 (unknown 3533727837 0 0) 0x7fe338001620 con 0x7fe348071bd0 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 >> v1:172.21.15.133:6789/0 conn(0x7fe348071bd0 legacy=0x7fe348071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 --2- 172.21.15.133:0/3356685109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe330003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7fe330004590 con 0x7fe330003a70 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.786+0000 7fe346ffd640 1 --2- 172.21.15.133:0/3356685109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe330003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:18.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.787+0000 7fe346ffd640 1 -- 172.21.15.133:0/3356685109 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe330003580 con 0x7fe330003a70 2024-08-16T02:36:18.789 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.788+0000 7fe346ffd640 1 --2- 172.21.15.133:0/3356685109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe330003ec0 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d950e0bd3469377 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:18.789 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe338007bf0 con 0x7fe330003a70 2024-08-16T02:36:18.789 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe33800f230 con 0x7fe330003a70 2024-08-16T02:36:18.789 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe345ffb640 1 -- 172.21.15.133:0/3356685109 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe33800f4d0 con 0x7fe330003a70 2024-08-16T02:36:18.790 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe34d82d640 1 -- 172.21.15.133:0/3356685109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 msgr2=0x7fe330003ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:18.790 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe34d82d640 1 --2- 172.21.15.133:0/3356685109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe330003ec0 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:18.791 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe34d82d640 1 -- 172.21.15.133:0/3356685109 shutdown_connections 2024-08-16T02:36:18.791 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe34d82d640 1 --2- 172.21.15.133:0/3356685109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe330003ec0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:18.791 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe34d82d640 1 --2- 172.21.15.133:0/3356685109 >> v2:172.21.15.133:3300/0 conn(0x7fe3480726f0 0x7fe3481385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:18.791 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.789+0000 7fe34d82d640 1 -- 172.21.15.133:0/3356685109 >> 172.21.15.133:0/3356685109 conn(0x7fe3480a2480 msgr2=0x7fe3480a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:18.791 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.790+0000 7fe34d82d640 1 -- 172.21.15.133:0/3356685109 shutdown_connections 2024-08-16T02:36:18.791 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.790+0000 7fe34d82d640 1 -- 172.21.15.133:0/3356685109 wait complete. 2024-08-16T02:36:18.791 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.790+0000 7fe34d82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.791 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-08-16T02:36:18.790+0000 7fbad01c5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.792 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-08-16T02:36:18.790+0000 7fbad01c5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.792 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.791+0000 7fe34d82d640 1 Processor -- start 2024-08-16T02:36:18.792 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.791+0000 7fe34d82d640 1 -- start start 2024-08-16T02:36:18.795 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.792+0000 7fe34d82d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe3481c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:18.795 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.792+0000 7fe34d82d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7fe348122c50 con 0x7fe330003a70 2024-08-16T02:36:18.795 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe346ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe3481c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:18.795 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe346ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe3481c74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:39720/0 (socket says 172.21.15.133:39720) 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe346ffd640 1 -- 172.21.15.133:0/4255360576 learned_addr learned my addr 172.21.15.133:0/4255360576 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe346ffd640 1 -- 172.21.15.133:0/4255360576 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3481267a0 con 0x7fe330003a70 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe346ffd640 1 --2- 172.21.15.133:0/4255360576 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe3481c74f0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe32ffff640 1 -- 172.21.15.133:0/4255360576 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe338001bf0 con 0x7fe330003a70 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe32ffff640 1 -- 172.21.15.133:0/4255360576 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe338003860 con 0x7fe330003a70 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe32ffff640 1 -- 172.21.15.133:0/4255360576 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe338003b00 con 0x7fe330003a70 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe348124240 con 0x7fe330003a70 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3481228f0 con 0x7fe330003a70 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe32ffff640 1 -- 172.21.15.133:0/4255360576 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7fe3380157a0 con 0x7fe330003a70 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.793+0000 7fe32ffff640 1 -- 172.21.15.133:0/4255360576 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (crc 0 0 0) 0x7fe33800fc80 con 0x7fe330003a70 2024-08-16T02:36:18.796 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.794+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe34806a020 con 0x7fe330003a70 2024-08-16T02:36:18.797 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.797+0000 7fe32ffff640 1 -- 172.21.15.133:0/4255360576 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7fe33800f3a0 con 0x7fe330003a70 2024-08-16T02:36:18.863 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.862+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fe348122a80 con 0x7fe330003a70 2024-08-16T02:36:18.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.863+0000 7fe32ffff640 1 -- 172.21.15.133:0/4255360576 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (crc 0 0 0) 0x7fe338003cb0 con 0x7fe330003a70 2024-08-16T02:36:18.864 INFO:teuthology.orchestra.run.smithi133.stderr:adjusted tunables profile to default 2024-08-16T02:36:18.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.864+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 msgr2=0x7fe3481c74f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:18.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.864+0000 7fe34d82d640 1 --2- 172.21.15.133:0/4255360576 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe3481c74f0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:18.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.864+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 shutdown_connections 2024-08-16T02:36:18.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.864+0000 7fe34d82d640 1 --2- 172.21.15.133:0/4255360576 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe330003a70 0x7fe3481c74f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:18.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.864+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 >> 172.21.15.133:0/4255360576 conn(0x7fe3480a2480 msgr2=0x7fe34811be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:18.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.864+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 shutdown_connections 2024-08-16T02:36:18.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.864+0000 7fe34d82d640 1 -- 172.21.15.133:0/4255360576 wait complete. 2024-08-16T02:36:18.876 INFO:tasks.ceph:check_enable_crimson: True 2024-08-16T02:36:18.876 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-08-16T02:36:18.876 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-08-16T02:36:18.982 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.976+0000 7f4b8eb52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.983 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.982+0000 7f4b8eb52640 1 Processor -- start 2024-08-16T02:36:18.983 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.982+0000 7f4b8eb52640 1 -- start start 2024-08-16T02:36:18.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.983+0000 7f4b8eb52640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4b88133120 0x7f4b88133530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:18.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b8eb52640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b8809bca0 con 0x7f4b88133a70 2024-08-16T02:36:18.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b8eb52640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f4b880a1fe0 con 0x7f4b88133120 2024-08-16T02:36:18.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b8c8ee640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4b88133120 0x7f4b88133530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:18.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b87fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f4b88133a70 0x7f4b8812fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:43046/0 (socket says 172.21.15.133:43046) 2024-08-16T02:36:18.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b87fff640 1 -- 172.21.15.133:0/2315837003 learned_addr learned my addr 172.21.15.133:0/2315837003 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:18.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b8c8ee640 1 -- 172.21.15.133:0/2315837003 >> v1:172.21.15.133:6789/0 conn(0x7f4b88133a70 legacy=0x7f4b8812fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:18.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b8c8ee640 1 -- 172.21.15.133:0/2315837003 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b880746c0 con 0x7f4b88133120 2024-08-16T02:36:18.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b8c8ee640 1 --2- 172.21.15.133:0/2315837003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b88133530 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df65b851a0bd36b server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:18.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b877fe640 1 -- 172.21.15.133:0/2315837003 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b7c009d90 con 0x7f4b88133120 2024-08-16T02:36:18.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b877fe640 1 -- 172.21.15.133:0/2315837003 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4b7c00e230 con 0x7f4b88133120 2024-08-16T02:36:18.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.984+0000 7f4b877fe640 1 -- 172.21.15.133:0/2315837003 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b7c00e4d0 con 0x7f4b88133120 2024-08-16T02:36:18.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.985+0000 7f4b8eb52640 1 -- 172.21.15.133:0/2315837003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 msgr2=0x7f4b88133530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:18.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.985+0000 7f4b8eb52640 1 --2- 172.21.15.133:0/2315837003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b88133530 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:18.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.985+0000 7f4b8eb52640 1 -- 172.21.15.133:0/2315837003 shutdown_connections 2024-08-16T02:36:18.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.985+0000 7f4b8eb52640 1 --2- 172.21.15.133:0/2315837003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b88133530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:18.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.985+0000 7f4b8eb52640 1 -- 172.21.15.133:0/2315837003 >> 172.21.15.133:0/2315837003 conn(0x7f4b880a21e0 msgr2=0x7f4b880a25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:18.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.985+0000 7f4b8eb52640 1 -- 172.21.15.133:0/2315837003 shutdown_connections 2024-08-16T02:36:18.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.985+0000 7f4b8eb52640 1 -- 172.21.15.133:0/2315837003 wait complete. 2024-08-16T02:36:18.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.985+0000 7f4b8eb52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:18.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.987+0000 7f4b8eb52640 1 Processor -- start 2024-08-16T02:36:18.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.987+0000 7f4b8eb52640 1 -- start start 2024-08-16T02:36:18.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b8eb52640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b8806f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b8eb52640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f4b88070e20 con 0x7f4b88133120 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b8c8ee640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b8806f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b8c8ee640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b8806f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:39732/0 (socket says 172.21.15.133:39732) 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b8c8ee640 1 -- 172.21.15.133:0/1103286641 learned_addr learned my addr 172.21.15.133:0/1103286641 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b8c8ee640 1 -- 172.21.15.133:0/1103286641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b88071430 con 0x7f4b88133120 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b8c8ee640 1 --2- 172.21.15.133:0/1103286641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b8806f9f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b857fa640 1 -- 172.21.15.133:0/1103286641 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b7c014620 con 0x7f4b88133120 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b857fa640 1 -- 172.21.15.133:0/1103286641 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4b7c014780 con 0x7f4b88133120 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.988+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b88122370 con 0x7f4b88133120 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.989+0000 7f4b857fa640 1 -- 172.21.15.133:0/1103286641 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b7c014a20 con 0x7f4b88133120 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.989+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b88121e20 con 0x7f4b88133120 2024-08-16T02:36:18.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.989+0000 7f4b857fa640 1 -- 172.21.15.133:0/1103286641 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f4b7c013400 con 0x7f4b88133120 2024-08-16T02:36:18.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.989+0000 7f4b857fa640 1 -- 172.21.15.133:0/1103286641 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (crc 0 0 0) 0x7f4b7c019040 con 0x7f4b88133120 2024-08-16T02:36:18.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.989+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b88077700 con 0x7f4b88133120 2024-08-16T02:36:18.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:18.992+0000 7f4b857fa640 1 -- 172.21.15.133:0/1103286641 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f4b7c013940 con 0x7f4b88133120 2024-08-16T02:36:19.068 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.067+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true} v 0) -- 0x7f4b880a1c60 con 0x7f4b88133120 2024-08-16T02:36:19.068 INFO:tasks.ceph.mon.a.smithi133.stderr:2024-08-16T02:36:19.067+0000 7fbd10df4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-08-16T02:36:19.068 INFO:tasks.ceph.mon.a.smithi133.stderr:Please be aware that this feature is experimental, untested, 2024-08-16T02:36:19.069 INFO:tasks.ceph.mon.a.smithi133.stderr:unsupported, and may result in data corruption, data loss, 2024-08-16T02:36:19.069 INFO:tasks.ceph.mon.a.smithi133.stderr:and/or irreparable damage to your cluster. Do not use 2024-08-16T02:36:19.069 INFO:tasks.ceph.mon.a.smithi133.stderr:feature with important data. 2024-08-16T02:36:19.069 INFO:tasks.ceph.mon.a.smithi133.stderr: 2024-08-16T02:36:19.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.864+0000 7f4b857fa640 1 -- 172.21.15.133:0/1103286641 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true}]=0 v3) ==== 99+0+0 (crc 0 0 0) 0x7f4b7c013bb0 con 0x7f4b88133120 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.865+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 msgr2=0x7f4b8806f9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.865+0000 7f4b8eb52640 1 --2- 172.21.15.133:0/1103286641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b8806f9f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.865+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 shutdown_connections 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.865+0000 7f4b8eb52640 1 --2- 172.21.15.133:0/1103286641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b88133120 0x7f4b8806f9f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.865+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 >> 172.21.15.133:0/1103286641 conn(0x7f4b880a21e0 msgr2=0x7f4b88131e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.865+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 shutdown_connections 2024-08-16T02:36:19.867 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:19.865+0000 7f4b8eb52640 1 -- 172.21.15.133:0/1103286641 wait complete. 2024-08-16T02:36:19.878 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-16T02:36:19.878 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:19.878 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-16T02:36:19.904 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:36:19.904 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-16T02:36:19.970 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:19.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-16T02:36:19.996 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-08-16T02:36:19.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-16T02:36:20.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 3108e258-24a8-4a0f-86a5-a3ee5db0a828 0 2024-08-16T02:36:20.212 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.205+0000 7f4cc5392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:20.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.211+0000 7f4cc5392640 1 Processor -- start 2024-08-16T02:36:20.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.212+0000 7f4cc5392640 1 -- start start 2024-08-16T02:36:20.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.212+0000 7f4cc5392640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4cc0118ea0 0x7f4cc01192b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:20.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.212+0000 7f4cc5392640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc0114b50 con 0x7f4cc0114030 2024-08-16T02:36:20.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.212+0000 7f4cc5392640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f4cc0114ce0 con 0x7f4cc0118ea0 2024-08-16T02:36:20.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.212+0000 7f4cbeffd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4cc0118ea0 0x7f4cc01192b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:20.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.212+0000 7f4cbeffd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4cc0118ea0 0x7f4cc01192b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.137:37278/0 (socket says 172.21.15.137:37278) 2024-08-16T02:36:20.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.212+0000 7f4cbeffd640 1 -- 172.21.15.137:0/484437228 learned_addr learned my addr 172.21.15.137:0/484437228 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-08-16T02:36:20.215 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.212+0000 7f4cbe7fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f4cc0114030 0x7f4cc0114440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.137:38832/0 (socket says 172.21.15.137:38832) 2024-08-16T02:36:20.215 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.213+0000 7f4cbeffd640 1 -- 172.21.15.137:0/484437228 >> v1:172.21.15.133:6789/0 conn(0x7f4cc0114030 legacy=0x7f4cc0114440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.215 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.213+0000 7f4cbeffd640 1 -- 172.21.15.137:0/484437228 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc0114e50 con 0x7f4cc0118ea0 2024-08-16T02:36:20.215 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.213+0000 7f4cbeffd640 1 --2- 172.21.15.137:0/484437228 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0118ea0 0x7f4cc01192b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee06c72d45e5604c server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:20.215 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.213+0000 7f4cbdffb640 1 -- 172.21.15.137:0/484437228 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4cb4009e50 con 0x7f4cc0118ea0 2024-08-16T02:36:20.215 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.213+0000 7f4cbdffb640 1 -- 172.21.15.137:0/484437228 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4cb400e230 con 0x7f4cc0118ea0 2024-08-16T02:36:20.215 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.213+0000 7f4cbdffb640 1 -- 172.21.15.137:0/484437228 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4cb400e4d0 con 0x7f4cc0118ea0 2024-08-16T02:36:20.217 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.214+0000 7f4cc5392640 1 -- 172.21.15.137:0/484437228 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0118ea0 msgr2=0x7f4cc01192b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.217 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.214+0000 7f4cc5392640 1 --2- 172.21.15.137:0/484437228 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0118ea0 0x7f4cc01192b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.214+0000 7f4cc5392640 1 -- 172.21.15.137:0/484437228 shutdown_connections 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.214+0000 7f4cc5392640 1 --2- 172.21.15.137:0/484437228 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0118ea0 0x7f4cc01192b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.214+0000 7f4cc5392640 1 -- 172.21.15.137:0/484437228 >> 172.21.15.137:0/484437228 conn(0x7f4cc006f840 msgr2=0x7f4cc006fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.214+0000 7f4cc5392640 1 -- 172.21.15.137:0/484437228 shutdown_connections 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.214+0000 7f4cc5392640 1 -- 172.21.15.137:0/484437228 wait complete. 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.214+0000 7f4cc5392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.215+0000 7f4cc5392640 1 Processor -- start 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.215+0000 7f4cc5392640 1 -- start start 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.215+0000 7f4cc5392640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0114030 0x7f4cc01a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.215+0000 7f4cc5392640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc01a9450 con 0x7f4cc0114030 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4cbeffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0114030 0x7f4cc01a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4cbeffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0114030 0x7f4cc01a8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.137:37282/0 (socket says 172.21.15.137:37282) 2024-08-16T02:36:20.218 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4cbeffd640 1 -- 172.21.15.137:0/4130340307 learned_addr learned my addr 172.21.15.137:0/4130340307 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4cbeffd640 1 -- 172.21.15.137:0/4130340307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc01a9590 con 0x7f4cc0114030 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4cbeffd640 1 --2- 172.21.15.137:0/4130340307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0114030 0x7f4cc01a8e80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4ca77fe640 1 -- 172.21.15.137:0/4130340307 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4cb400eea0 con 0x7f4cc0114030 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4ca77fe640 1 -- 172.21.15.137:0/4130340307 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4cb4015420 con 0x7f4cc0114030 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4ca77fe640 1 -- 172.21.15.137:0/4130340307 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4cb40156c0 con 0x7f4cc0114030 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc01a97e0 con 0x7f4cc0114030 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.216+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc01ae6e0 con 0x7f4cc0114030 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.217+0000 7f4ca77fe640 1 -- 172.21.15.137:0/4130340307 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f4cb4015c00 con 0x7f4cc0114030 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.217+0000 7f4ca77fe640 1 -- 172.21.15.137:0/4130340307 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 747+0+0 (crc 0 0 0) 0x7f4cb40116d0 con 0x7f4cc0114030 2024-08-16T02:36:20.219 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.218+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc0077700 con 0x7f4cc0114030 2024-08-16T02:36:20.221 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.220+0000 7f4ca77fe640 1 -- 172.21.15.137:0/4130340307 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f4cb4011930 con 0x7f4cc0114030 2024-08-16T02:36:20.316 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.315+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3108e258-24a8-4a0f-86a5-a3ee5db0a828", "id": 0} v 0) -- 0x7f4cc01200a0 con 0x7f4cc0114030 2024-08-16T02:36:20.318 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.317+0000 7f4ca77fe640 1 -- 172.21.15.137:0/4130340307 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3108e258-24a8-4a0f-86a5-a3ee5db0a828", "id": 0}]=0 v4) ==== 112+0+2 (crc 0 0 0) 0x7f4cb4011ad0 con 0x7f4cc0114030 2024-08-16T02:36:20.318 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-08-16T02:36:20.319 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.318+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0114030 msgr2=0x7f4cc01a8e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.319 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.318+0000 7f4cc5392640 1 --2- 172.21.15.137:0/4130340307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0114030 0x7f4cc01a8e80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.319 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.318+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 shutdown_connections 2024-08-16T02:36:20.319 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.318+0000 7f4cc5392640 1 --2- 172.21.15.137:0/4130340307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4cc0114030 0x7f4cc01a8e80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.319 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.318+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 >> 172.21.15.137:0/4130340307 conn(0x7f4cc006f840 msgr2=0x7f4cc0117af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:20.320 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.318+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 shutdown_connections 2024-08-16T02:36:20.320 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.318+0000 7f4cc5392640 1 -- 172.21.15.137:0/4130340307 wait complete. 2024-08-16T02:36:20.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 103b6a98-7079-4079-a37b-4216a77b9205 1 2024-08-16T02:36:20.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.431+0000 7f8c85d12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:20.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.438+0000 7f8c85d12640 1 Processor -- start 2024-08-16T02:36:20.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.438+0000 7f8c85d12640 1 -- start start 2024-08-16T02:36:20.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.438+0000 7f8c85d12640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f8c80118e90 0x7f8c801192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:20.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.438+0000 7f8c85d12640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c80114b90 con 0x7f8c80114070 2024-08-16T02:36:20.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.438+0000 7f8c85d12640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f8c80114d20 con 0x7f8c80118e90 2024-08-16T02:36:20.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.438+0000 7f8c7f7fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f8c80118e90 0x7f8c801192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:20.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.438+0000 7f8c7effd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f8c80114070 0x7f8c80114480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.137:38838/0 (socket says 172.21.15.137:38838) 2024-08-16T02:36:20.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.438+0000 7f8c7effd640 1 -- 172.21.15.137:0/2748104048 learned_addr learned my addr 172.21.15.137:0/2748104048 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-08-16T02:36:20.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.439+0000 7f8c7f7fe640 1 -- 172.21.15.137:0/2748104048 >> v1:172.21.15.133:6789/0 conn(0x7f8c80114070 legacy=0x7f8c80114480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.439+0000 7f8c7f7fe640 1 -- 172.21.15.137:0/2748104048 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c80114e90 con 0x7f8c80118e90 2024-08-16T02:36:20.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.439+0000 7f8c7f7fe640 1 --2- 172.21.15.137:0/2748104048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80118e90 0x7f8c801192a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6355f358b2604f38 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:20.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.439+0000 7f8c7e7fc640 1 -- 172.21.15.137:0/2748104048 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8c70009d90 con 0x7f8c80118e90 2024-08-16T02:36:20.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.439+0000 7f8c7e7fc640 1 -- 172.21.15.137:0/2748104048 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f8c7000e230 con 0x7f8c80118e90 2024-08-16T02:36:20.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.439+0000 7f8c7e7fc640 1 -- 172.21.15.137:0/2748104048 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8c7000e4d0 con 0x7f8c80118e90 2024-08-16T02:36:20.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.440+0000 7f8c85d12640 1 -- 172.21.15.137:0/2748104048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80118e90 msgr2=0x7f8c801192a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.440+0000 7f8c85d12640 1 --2- 172.21.15.137:0/2748104048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80118e90 0x7f8c801192a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.440+0000 7f8c85d12640 1 -- 172.21.15.137:0/2748104048 shutdown_connections 2024-08-16T02:36:20.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.440+0000 7f8c85d12640 1 --2- 172.21.15.137:0/2748104048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80118e90 0x7f8c801192a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.440+0000 7f8c85d12640 1 -- 172.21.15.137:0/2748104048 >> 172.21.15.137:0/2748104048 conn(0x7f8c8006f810 msgr2=0x7f8c8006fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:20.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.440+0000 7f8c85d12640 1 -- 172.21.15.137:0/2748104048 shutdown_connections 2024-08-16T02:36:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.440+0000 7f8c85d12640 1 -- 172.21.15.137:0/2748104048 wait complete. 2024-08-16T02:36:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.440+0000 7f8c85d12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.441+0000 7f8c85d12640 1 Processor -- start 2024-08-16T02:36:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.441+0000 7f8c85d12640 1 -- start start 2024-08-16T02:36:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.441+0000 7f8c85d12640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80114070 0x7f8c801a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.441+0000 7f8c85d12640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f8c801a93c0 con 0x7f8c80114070 2024-08-16T02:36:20.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c7f7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80114070 0x7f8c801a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:20.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c7f7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80114070 0x7f8c801a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.137:37298/0 (socket says 172.21.15.137:37298) 2024-08-16T02:36:20.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c7f7fe640 1 -- 172.21.15.137:0/2749713712 learned_addr learned my addr 172.21.15.137:0/2749713712 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-08-16T02:36:20.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c7f7fe640 1 -- 172.21.15.137:0/2749713712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c801a9500 con 0x7f8c80114070 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c7f7fe640 1 --2- 172.21.15.137:0/2749713712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80114070 0x7f8c801a8df0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c67fff640 1 -- 172.21.15.137:0/2749713712 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8c70014620 con 0x7f8c80114070 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c67fff640 1 -- 172.21.15.137:0/2749713712 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f8c70014780 con 0x7f8c80114070 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c801a9750 con 0x7f8c80114070 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c67fff640 1 -- 172.21.15.137:0/2749713712 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8c70014a20 con 0x7f8c80114070 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.442+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c801ae650 con 0x7f8c80114070 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.443+0000 7f8c67fff640 1 -- 172.21.15.137:0/2749713712 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f8c70013400 con 0x7f8c80114070 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.443+0000 7f8c67fff640 1 -- 172.21.15.137:0/2749713712 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 862+0+0 (crc 0 0 0) 0x7f8c70019040 con 0x7f8c80114070 2024-08-16T02:36:20.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.444+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c80077700 con 0x7f8c80114070 2024-08-16T02:36:20.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.446+0000 7f8c67fff640 1 -- 172.21.15.137:0/2749713712 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f8c70013840 con 0x7f8c80114070 2024-08-16T02:36:20.521 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.521+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd new", "uuid": "103b6a98-7079-4079-a37b-4216a77b9205", "id": 1} v 0) -- 0x7f8c80120090 con 0x7f8c80114070 2024-08-16T02:36:20.523 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.522+0000 7f8c67fff640 1 -- 172.21.15.137:0/2749713712 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "103b6a98-7079-4079-a37b-4216a77b9205", "id": 1}]=0 v5) ==== 112+0+2 (crc 0 0 0) 0x7f8c70013b30 con 0x7f8c80114070 2024-08-16T02:36:20.524 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-08-16T02:36:20.524 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.523+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80114070 msgr2=0x7f8c801a8df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.525 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.523+0000 7f8c85d12640 1 --2- 172.21.15.137:0/2749713712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80114070 0x7f8c801a8df0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.525 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.523+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 shutdown_connections 2024-08-16T02:36:20.525 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.523+0000 7f8c85d12640 1 --2- 172.21.15.137:0/2749713712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8c80114070 0x7f8c801a8df0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.525 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.523+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 >> 172.21.15.137:0/2749713712 conn(0x7f8c8006f810 msgr2=0x7f8c80117a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:20.525 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.523+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 shutdown_connections 2024-08-16T02:36:20.525 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.523+0000 7f8c85d12640 1 -- 172.21.15.137:0/2749713712 wait complete. 2024-08-16T02:36:20.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 14d9be38-a2e8-482d-96b8-48359bed89bd 2 2024-08-16T02:36:20.643 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.637+0000 7f119b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:20.644 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.643+0000 7f119b8ad640 1 Processor -- start 2024-08-16T02:36:20.644 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.643+0000 7f119b8ad640 1 -- start start 2024-08-16T02:36:20.645 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.643+0000 7f119b8ad640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1194118e90 0x7f11941192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:20.645 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.643+0000 7f119b8ad640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1194114b90 con 0x7f1194114070 2024-08-16T02:36:20.645 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.643+0000 7f119b8ad640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f1194114d20 con 0x7f1194118e90 2024-08-16T02:36:20.645 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.644+0000 7f1199649640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1194118e90 0x7f11941192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:20.645 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.644+0000 7f1199649640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1194118e90 0x7f11941192a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.137:37310/0 (socket says 172.21.15.137:37310) 2024-08-16T02:36:20.645 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.644+0000 7f1199649640 1 -- 172.21.15.137:0/3476784049 learned_addr learned my addr 172.21.15.137:0/3476784049 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-08-16T02:36:20.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.644+0000 7f1198e48640 1 --1- 172.21.15.137:0/3476784049 >> v1:172.21.15.133:6789/0 conn(0x7f1194114070 0x7f1194114480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.137:38846/0 (socket says 172.21.15.137:38846) 2024-08-16T02:36:20.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.644+0000 7f1199649640 1 -- 172.21.15.137:0/3476784049 >> v1:172.21.15.133:6789/0 conn(0x7f1194114070 legacy=0x7f1194114480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.644+0000 7f1199649640 1 -- 172.21.15.137:0/3476784049 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1194114e90 con 0x7f1194118e90 2024-08-16T02:36:20.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.644+0000 7f1199649640 1 --2- 172.21.15.137:0/3476784049 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194118e90 0x7f11941192a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86afa7f0d1f36a9 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:20.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.644+0000 7f118bfff640 1 -- 172.21.15.137:0/3476784049 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1184009e50 con 0x7f1194118e90 2024-08-16T02:36:20.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f118bfff640 1 -- 172.21.15.137:0/3476784049 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f118400e230 con 0x7f1194118e90 2024-08-16T02:36:20.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f118bfff640 1 -- 172.21.15.137:0/3476784049 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f118400e4d0 con 0x7f1194118e90 2024-08-16T02:36:20.647 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f119b8ad640 1 -- 172.21.15.137:0/3476784049 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194118e90 msgr2=0x7f11941192a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.647 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f119b8ad640 1 --2- 172.21.15.137:0/3476784049 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194118e90 0x7f11941192a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.647 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f119b8ad640 1 -- 172.21.15.137:0/3476784049 shutdown_connections 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f119b8ad640 1 --2- 172.21.15.137:0/3476784049 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194118e90 0x7f11941192a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f119b8ad640 1 -- 172.21.15.137:0/3476784049 >> 172.21.15.137:0/3476784049 conn(0x7f119406f810 msgr2=0x7f119406fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f119b8ad640 1 -- 172.21.15.137:0/3476784049 shutdown_connections 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.645+0000 7f119b8ad640 1 -- 172.21.15.137:0/3476784049 wait complete. 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.646+0000 7f119b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.646+0000 7f119b8ad640 1 Processor -- start 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.646+0000 7f119b8ad640 1 -- start start 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.647+0000 7f119b8ad640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194114070 0x7f11941a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:20.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.647+0000 7f119b8ad640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f11941a93c0 con 0x7f1194114070 2024-08-16T02:36:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.647+0000 7f1199649640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194114070 0x7f11941a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.647+0000 7f1199649640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194114070 0x7f11941a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.137:37318/0 (socket says 172.21.15.137:37318) 2024-08-16T02:36:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.647+0000 7f1199649640 1 -- 172.21.15.137:0/1715606459 learned_addr learned my addr 172.21.15.137:0/1715606459 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-08-16T02:36:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.647+0000 7f1199649640 1 -- 172.21.15.137:0/1715606459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11941a9500 con 0x7f1194114070 2024-08-16T02:36:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.648+0000 7f1199649640 1 --2- 172.21.15.137:0/1715606459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194114070 0x7f11941a8df0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.648+0000 7f1189ffb640 1 -- 172.21.15.137:0/1715606459 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f118400eea0 con 0x7f1194114070 2024-08-16T02:36:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.648+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11941a9750 con 0x7f1194114070 2024-08-16T02:36:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.648+0000 7f1189ffb640 1 -- 172.21.15.137:0/1715606459 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1184015420 con 0x7f1194114070 2024-08-16T02:36:20.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.648+0000 7f1189ffb640 1 -- 172.21.15.137:0/1715606459 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f11840156c0 con 0x7f1194114070 2024-08-16T02:36:20.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.648+0000 7f1189ffb640 1 -- 172.21.15.137:0/1715606459 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f1184015bb0 con 0x7f1194114070 2024-08-16T02:36:20.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.648+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11941ae650 con 0x7f1194114070 2024-08-16T02:36:20.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.648+0000 7f1189ffb640 1 -- 172.21.15.137:0/1715606459 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 977+0+0 (crc 0 0 0) 0x7f11840119a0 con 0x7f1194114070 2024-08-16T02:36:20.651 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.649+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1194120090 con 0x7f1194114070 2024-08-16T02:36:20.652 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.652+0000 7f1189ffb640 1 -- 172.21.15.137:0/1715606459 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f1184011560 con 0x7f1194114070 2024-08-16T02:36:20.728 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.727+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd new", "uuid": "14d9be38-a2e8-482d-96b8-48359bed89bd", "id": 2} v 0) -- 0x7f1194120240 con 0x7f1194114070 2024-08-16T02:36:20.729 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.729+0000 7f1189ffb640 1 -- 172.21.15.137:0/1715606459 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "14d9be38-a2e8-482d-96b8-48359bed89bd", "id": 2}]=0 v6) ==== 112+0+2 (crc 0 0 0) 0x7f1184011780 con 0x7f1194114070 2024-08-16T02:36:20.729 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-08-16T02:36:20.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.729+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194114070 msgr2=0x7f11941a8df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.729+0000 7f119b8ad640 1 --2- 172.21.15.137:0/1715606459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194114070 0x7f11941a8df0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.730+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 shutdown_connections 2024-08-16T02:36:20.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.730+0000 7f119b8ad640 1 --2- 172.21.15.137:0/1715606459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1194114070 0x7f11941a8df0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.730+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 >> 172.21.15.137:0/1715606459 conn(0x7f119406f810 msgr2=0x7f1194117a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:20.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.730+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 shutdown_connections 2024-08-16T02:36:20.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.730+0000 7f119b8ad640 1 -- 172.21.15.137:0/1715606459 wait complete. 2024-08-16T02:36:20.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new d03f5fd2-ce08-4ea9-8390-2efa9b037a9e 3 2024-08-16T02:36:20.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.844+0000 7f1548a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:20.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.850+0000 7f1548a24640 1 Processor -- start 2024-08-16T02:36:20.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.850+0000 7f1548a24640 1 -- start start 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.851+0000 7f1548a24640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1544118e90 0x7f15441192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.851+0000 7f1548a24640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1544114b90 con 0x7f1544114070 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.851+0000 7f1548a24640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f1544114d20 con 0x7f1544118e90 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.851+0000 7f154259c640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1544118e90 0x7f15441192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.851+0000 7f154259c640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1544118e90 0x7f15441192a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.137:37332/0 (socket says 172.21.15.137:37332) 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.851+0000 7f154259c640 1 -- 172.21.15.137:0/27549304 learned_addr learned my addr 172.21.15.137:0/27549304 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.851+0000 7f1541d9b640 1 --1- 172.21.15.137:0/27549304 >> v1:172.21.15.133:6789/0 conn(0x7f1544114070 0x7f1544114480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.137:38852/0 (socket says 172.21.15.137:38852) 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.852+0000 7f154259c640 1 -- 172.21.15.137:0/27549304 >> v1:172.21.15.133:6789/0 conn(0x7f1544114070 legacy=0x7f1544114480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.852+0000 7f154259c640 1 -- 172.21.15.137:0/27549304 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1544114e90 con 0x7f1544118e90 2024-08-16T02:36:20.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.852+0000 7f154259c640 1 --2- 172.21.15.137:0/27549304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544118e90 0x7f15441192a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89879b81a5c8908c server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:20.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.852+0000 7f154159a640 1 -- 172.21.15.137:0/27549304 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1534009e50 con 0x7f1544118e90 2024-08-16T02:36:20.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.852+0000 7f154159a640 1 -- 172.21.15.137:0/27549304 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f153400e230 con 0x7f1544118e90 2024-08-16T02:36:20.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.852+0000 7f154159a640 1 -- 172.21.15.137:0/27549304 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f153400e4d0 con 0x7f1544118e90 2024-08-16T02:36:20.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.853+0000 7f1548a24640 1 -- 172.21.15.137:0/27549304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544118e90 msgr2=0x7f15441192a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.853+0000 7f1548a24640 1 --2- 172.21.15.137:0/27549304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544118e90 0x7f15441192a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.853+0000 7f1548a24640 1 -- 172.21.15.137:0/27549304 shutdown_connections 2024-08-16T02:36:20.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.853+0000 7f1548a24640 1 --2- 172.21.15.137:0/27549304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544118e90 0x7f15441192a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.853+0000 7f1548a24640 1 -- 172.21.15.137:0/27549304 >> 172.21.15.137:0/27549304 conn(0x7f154406f810 msgr2=0x7f154406fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.853+0000 7f1548a24640 1 -- 172.21.15.137:0/27549304 shutdown_connections 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.853+0000 7f1548a24640 1 -- 172.21.15.137:0/27549304 wait complete. 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.853+0000 7f1548a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.854+0000 7f1548a24640 1 Processor -- start 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.854+0000 7f1548a24640 1 -- start start 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.854+0000 7f1548a24640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544114070 0x7f15441a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.854+0000 7f1548a24640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f15441a93c0 con 0x7f1544114070 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f154259c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544114070 0x7f15441a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f154259c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544114070 0x7f15441a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.137:37336/0 (socket says 172.21.15.137:37336) 2024-08-16T02:36:20.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f154259c640 1 -- 172.21.15.137:0/2082121972 learned_addr learned my addr 172.21.15.137:0/2082121972 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-08-16T02:36:20.857 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f154259c640 1 -- 172.21.15.137:0/2082121972 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15441a9500 con 0x7f1544114070 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f154259c640 1 --2- 172.21.15.137:0/2082121972 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544114070 0x7f15441a8df0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f1532ffd640 1 -- 172.21.15.137:0/2082121972 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f153400eea0 con 0x7f1544114070 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f1532ffd640 1 -- 172.21.15.137:0/2082121972 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1534015420 con 0x7f1544114070 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15441a9750 con 0x7f1544114070 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f1532ffd640 1 -- 172.21.15.137:0/2082121972 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f15340156c0 con 0x7f1544114070 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.855+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15441ae650 con 0x7f1544114070 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.856+0000 7f1532ffd640 1 -- 172.21.15.137:0/2082121972 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f1534015c50 con 0x7f1544114070 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.856+0000 7f1532ffd640 1 -- 172.21.15.137:0/2082121972 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1092+0+0 (crc 0 0 0) 0x7f1534014a90 con 0x7f1544114070 2024-08-16T02:36:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.856+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1544077700 con 0x7f1544114070 2024-08-16T02:36:20.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.859+0000 7f1532ffd640 1 -- 172.21.15.137:0/2082121972 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f15340143b0 con 0x7f1544114070 2024-08-16T02:36:20.934 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.933+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d03f5fd2-ce08-4ea9-8390-2efa9b037a9e", "id": 3} v 0) -- 0x7f1544118e90 con 0x7f1544114070 2024-08-16T02:36:20.936 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.936+0000 7f1532ffd640 1 -- 172.21.15.137:0/2082121972 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d03f5fd2-ce08-4ea9-8390-2efa9b037a9e", "id": 3}]=0 v7) ==== 112+0+2 (crc 0 0 0) 0x7f15340145d0 con 0x7f1544114070 2024-08-16T02:36:20.937 INFO:teuthology.orchestra.run.smithi137.stdout:3 2024-08-16T02:36:20.937 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.937+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544114070 msgr2=0x7f15441a8df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:20.938 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.937+0000 7f1548a24640 1 --2- 172.21.15.137:0/2082121972 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544114070 0x7f15441a8df0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.938 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.937+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 shutdown_connections 2024-08-16T02:36:20.938 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.937+0000 7f1548a24640 1 --2- 172.21.15.137:0/2082121972 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1544114070 0x7f15441a8df0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:20.938 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.937+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 >> 172.21.15.137:0/2082121972 conn(0x7f154406f810 msgr2=0x7f1544117a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:20.938 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.937+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 shutdown_connections 2024-08-16T02:36:20.938 INFO:teuthology.orchestra.run.smithi137.stderr:2024-08-16T02:36:20.937+0000 7f1548a24640 1 -- 172.21.15.137:0/2082121972 wait complete. 2024-08-16T02:36:20.948 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-08-16T02:36:20.983 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-16T02:36:20.983 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-08-16T02:36:21.026 INFO:tasks.ceph.osd.0:Started 2024-08-16T02:36:21.026 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-08-16T02:36:21.057 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-16T02:36:21.057 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-08-16T02:36:21.100 INFO:tasks.ceph.osd.1:Started 2024-08-16T02:36:21.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-08-16T02:36:21.132 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-16T02:36:21.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-08-16T02:36:21.174 INFO:tasks.ceph.osd.2:Started 2024-08-16T02:36:21.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-08-16T02:36:21.204 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-16T02:36:21.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-08-16T02:36:21.246 INFO:tasks.ceph.osd.3:Started 2024-08-16T02:36:21.246 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:36:21.362 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.357+0000 7ff42f26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:21.362 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.362+0000 7ff42f26d640 1 Processor -- start 2024-08-16T02:36:21.363 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.362+0000 7ff42f26d640 1 -- start start 2024-08-16T02:36:21.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.363+0000 7ff42f26d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7ff4280a0950 0x7ff42813a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:21.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.363+0000 7ff42f26d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4280a14c0 con 0x7ff42809fe30 2024-08-16T02:36:21.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.363+0000 7ff42f26d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7ff42806f380 con 0x7ff4280a0950 2024-08-16T02:36:21.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.363+0000 7ff42c808640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7ff4280a0950 0x7ff42813a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:21.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff42c808640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7ff4280a0950 0x7ff42813a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43614/0 (socket says 172.21.15.133:43614) 2024-08-16T02:36:21.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff42c808640 1 -- 172.21.15.133:0/531118326 learned_addr learned my addr 172.21.15.133:0/531118326 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:21.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff42d009640 1 --1- 172.21.15.133:0/531118326 >> v1:172.21.15.133:6789/0 conn(0x7ff42809fe30 0x7ff4280a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:36358/0 (socket says 172.21.15.133:36358) 2024-08-16T02:36:21.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff42c808640 1 -- 172.21.15.133:0/531118326 >> v1:172.21.15.133:6789/0 conn(0x7ff42809fe30 legacy=0x7ff4280a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:21.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff42c808640 1 -- 172.21.15.133:0/531118326 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4280a2130 con 0x7ff4280a0950 2024-08-16T02:36:21.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff42c808640 1 --2- 172.21.15.133:0/531118326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4280a0950 0x7ff42813a7e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb721af012f0b04 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:21.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff427fff640 1 -- 172.21.15.133:0/531118326 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff41c009e50 con 0x7ff4280a0950 2024-08-16T02:36:21.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff427fff640 1 -- 172.21.15.133:0/531118326 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff41c00e230 con 0x7ff4280a0950 2024-08-16T02:36:21.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.364+0000 7ff427fff640 1 -- 172.21.15.133:0/531118326 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff41c00e4d0 con 0x7ff4280a0950 2024-08-16T02:36:21.367 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.365+0000 7ff42f26d640 1 -- 172.21.15.133:0/531118326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4280a0950 msgr2=0x7ff42813a7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:21.367 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.365+0000 7ff42f26d640 1 --2- 172.21.15.133:0/531118326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4280a0950 0x7ff42813a7e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:21.367 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.365+0000 7ff42f26d640 1 -- 172.21.15.133:0/531118326 shutdown_connections 2024-08-16T02:36:21.367 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.365+0000 7ff42f26d640 1 --2- 172.21.15.133:0/531118326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4280a0950 0x7ff42813a7e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:21.367 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.365+0000 7ff42f26d640 1 -- 172.21.15.133:0/531118326 >> 172.21.15.133:0/531118326 conn(0x7ff42806caf0 msgr2=0x7ff42806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:21.367 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.365+0000 7ff42f26d640 1 -- 172.21.15.133:0/531118326 shutdown_connections 2024-08-16T02:36:21.367 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.365+0000 7ff42f26d640 1 -- 172.21.15.133:0/531118326 wait complete. 2024-08-16T02:36:21.367 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.365+0000 7ff42f26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:21.368 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.367+0000 7ff42f26d640 1 Processor -- start 2024-08-16T02:36:21.368 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.367+0000 7ff42f26d640 1 -- start start 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff42f26d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff42809fe30 0x7ff428124c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff42f26d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7ff4281266b0 con 0x7ff42809fe30 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff42d009640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff42809fe30 0x7ff428124c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff42d009640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff42809fe30 0x7ff428124c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43616/0 (socket says 172.21.15.133:43616) 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff42d009640 1 -- 172.21.15.133:0/2759223307 learned_addr learned my addr 172.21.15.133:0/2759223307 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff42d009640 1 -- 172.21.15.133:0/2759223307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42806fcf0 con 0x7ff42809fe30 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff42d009640 1 --2- 172.21.15.133:0/2759223307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff42809fe30 0x7ff428124c30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff425ffb640 1 -- 172.21.15.133:0/2759223307 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff4200059a0 con 0x7ff42809fe30 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff425ffb640 1 -- 172.21.15.133:0/2759223307 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff420005b00 con 0x7ff42809fe30 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff425ffb640 1 -- 172.21.15.133:0/2759223307 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff420005da0 con 0x7ff42809fe30 2024-08-16T02:36:21.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.368+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff428127dd0 con 0x7ff42809fe30 2024-08-16T02:36:21.371 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.369+0000 7ff425ffb640 1 -- 172.21.15.133:0/2759223307 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7ff420007440 con 0x7ff42809fe30 2024-08-16T02:36:21.371 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.369+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff42813a1e0 con 0x7ff42809fe30 2024-08-16T02:36:21.371 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.369+0000 7ff425ffb640 1 -- 172.21.15.133:0/2759223307 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1207+0+0 (crc 0 0 0) 0x7ff42000e1c0 con 0x7ff42809fe30 2024-08-16T02:36:21.371 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.369+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4280a0240 con 0x7ff42809fe30 2024-08-16T02:36:21.372 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.372+0000 7ff425ffb640 1 -- 172.21.15.133:0/2759223307 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7ff42000c8a0 con 0x7ff42809fe30 2024-08-16T02:36:21.438 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.437+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff428128c10 con 0x7ff42809fe30 2024-08-16T02:36:21.439 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.438+0000 7ff425ffb640 1 -- 172.21.15.133:0/2759223307 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) ==== 74+0+4152 (crc 0 0 0) 0x7ff42000caf0 con 0x7ff42809fe30 2024-08-16T02:36:21.439 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:21.439 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":7,"fsid":"7a5c695b-6aa6-4555-8e56-4a247af82439","created":"2024-08-16T02:36:18.775904+0000","modified":"2024-08-16T02:36:20.935538+0000","last_up_change":"0.000000","last_in_change":"2024-08-16T02:36:20.935538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3108e258-24a8-4a0f-86a5-a3ee5db0a828","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"103b6a98-7079-4079-a37b-4216a77b9205","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"14d9be38-a2e8-482d-96b8-48359bed89bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d03f5fd2-ce08-4ea9-8390-2efa9b037a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:36:21.440 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.439+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff42809fe30 msgr2=0x7ff428124c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:21.440 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.439+0000 7ff42f26d640 1 --2- 172.21.15.133:0/2759223307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff42809fe30 0x7ff428124c30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:21.440 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.439+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 shutdown_connections 2024-08-16T02:36:21.440 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.439+0000 7ff42f26d640 1 --2- 172.21.15.133:0/2759223307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff42809fe30 0x7ff428124c30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:21.440 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.439+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 >> 172.21.15.133:0/2759223307 conn(0x7ff42806caf0 msgr2=0x7ff4280a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:21.440 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.439+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 shutdown_connections 2024-08-16T02:36:21.441 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:21.439+0000 7ff42f26d640 1 -- 172.21.15.133:0/2759223307 wait complete. 2024-08-16T02:36:21.451 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-16T02:36:21.451 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-16T02:36:21.854 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-16T02:36:21.960 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.955+0000 7f2ac36ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:21.961 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.960+0000 7f2ac36ad640 1 Processor -- start 2024-08-16T02:36:21.961 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.960+0000 7f2ac36ad640 1 -- start start 2024-08-16T02:36:21.963 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.961+0000 7f2ac36ad640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f2abc0a2110 0x7f2abc13a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:21.963 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.961+0000 7f2ac36ad640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2abc06ed70 con 0x7f2abc0a15f0 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.961+0000 7f2ac36ad640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f2abc06f820 con 0x7f2abc0a2110 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ac0c48640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f2abc0a2110 0x7f2abc13a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ac1449640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2abc0a15f0 0x7f2abc0a1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:36368/0 (socket says 172.21.15.133:36368) 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ac1449640 1 -- 172.21.15.133:0/2827320471 learned_addr learned my addr 172.21.15.133:0/2827320471 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ac0c48640 1 -- 172.21.15.133:0/2827320471 >> v1:172.21.15.133:6789/0 conn(0x7f2abc0a15f0 legacy=0x7f2abc0a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ab3fff640 1 -- 172.21.15.133:0/2827320471 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204881986 0 0) 0x7f2abc06ed70 con 0x7f2abc0a15f0 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ac0c48640 1 -- 172.21.15.133:0/2827320471 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc0a4340 con 0x7f2abc0a2110 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ac0c48640 1 --2- 172.21.15.133:0/2827320471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a2110 0x7f2abc13a670 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dfc155e89cfe600 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:21.964 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ab3fff640 1 -- 172.21.15.133:0/2827320471 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2aac009d90 con 0x7f2abc0a2110 2024-08-16T02:36:21.965 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ab3fff640 1 -- 172.21.15.133:0/2827320471 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2aac00e230 con 0x7f2abc0a2110 2024-08-16T02:36:21.965 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.962+0000 7f2ab3fff640 1 -- 172.21.15.133:0/2827320471 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2aac00e4d0 con 0x7f2abc0a2110 2024-08-16T02:36:21.966 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.963+0000 7f2ac36ad640 1 -- 172.21.15.133:0/2827320471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a2110 msgr2=0x7f2abc13a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:21.966 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.963+0000 7f2ac36ad640 1 --2- 172.21.15.133:0/2827320471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a2110 0x7f2abc13a670 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:21.966 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.963+0000 7f2ac36ad640 1 -- 172.21.15.133:0/2827320471 shutdown_connections 2024-08-16T02:36:21.966 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.963+0000 7f2ac36ad640 1 --2- 172.21.15.133:0/2827320471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a2110 0x7f2abc13a670 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:21.966 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.963+0000 7f2ac36ad640 1 -- 172.21.15.133:0/2827320471 >> 172.21.15.133:0/2827320471 conn(0x7f2abc0a4e90 msgr2=0x7f2abc09fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:21.966 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.964+0000 7f2ac36ad640 1 -- 172.21.15.133:0/2827320471 shutdown_connections 2024-08-16T02:36:21.967 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.964+0000 7f2ac36ad640 1 -- 172.21.15.133:0/2827320471 wait complete. 2024-08-16T02:36:21.967 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.964+0000 7f2ac36ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:21.967 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.965+0000 7f2ac36ad640 1 Processor -- start 2024-08-16T02:36:21.967 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.965+0000 7f2ac36ad640 1 -- start start 2024-08-16T02:36:21.968 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.966+0000 7f2ac36ad640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a15f0 0x7f2abc123670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.966+0000 7f2ac36ad640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc125650 con 0x7f2abc0a15f0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.966+0000 7f2ac1449640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a15f0 0x7f2abc123670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.966+0000 7f2ac1449640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a15f0 0x7f2abc123670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43624/0 (socket says 172.21.15.133:43624) 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.966+0000 7f2ac1449640 1 -- 172.21.15.133:0/1898745858 learned_addr learned my addr 172.21.15.133:0/1898745858 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ac1449640 1 -- 172.21.15.133:0/1898745858 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc124680 con 0x7f2abc0a15f0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ac1449640 1 --2- 172.21.15.133:0/1898745858 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a15f0 0x7f2abc123670 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ab1ffb640 1 -- 172.21.15.133:0/1898745858 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2ab8004b80 con 0x7f2abc0a15f0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ab1ffb640 1 -- 172.21.15.133:0/1898745858 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2ab8004ce0 con 0x7f2abc0a15f0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ab1ffb640 1 -- 172.21.15.133:0/1898745858 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2ab8007120 con 0x7f2abc0a15f0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc122bd0 con 0x7f2abc0a15f0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ab1ffb640 1 -- 172.21.15.133:0/1898745858 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f2ab800c520 con 0x7f2abc0a15f0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc122e30 con 0x7f2abc0a15f0 2024-08-16T02:36:21.969 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.967+0000 7f2ab1ffb640 1 -- 172.21.15.133:0/1898745858 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1207+0+0 (crc 0 0 0) 0x7f2ab8007660 con 0x7f2abc0a15f0 2024-08-16T02:36:21.970 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.968+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc126aa0 con 0x7f2abc0a15f0 2024-08-16T02:36:21.971 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:21.970+0000 7f2ab1ffb640 1 -- 172.21.15.133:0/1898745858 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f2ab800e1c0 con 0x7f2abc0a15f0 2024-08-16T02:36:22.035 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.034+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2abc0a1a00 con 0x7f2abc0a15f0 2024-08-16T02:36:22.036 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.035+0000 7f2ab1ffb640 1 -- 172.21.15.133:0/1898745858 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) ==== 74+0+4152 (crc 0 0 0) 0x7f2ab800ca50 con 0x7f2abc0a15f0 2024-08-16T02:36:22.036 INFO:teuthology.misc.health.smithi133.stdout: 2024-08-16T02:36:22.036 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":7,"fsid":"7a5c695b-6aa6-4555-8e56-4a247af82439","created":"2024-08-16T02:36:18.775904+0000","modified":"2024-08-16T02:36:20.935538+0000","last_up_change":"0.000000","last_in_change":"2024-08-16T02:36:20.935538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3108e258-24a8-4a0f-86a5-a3ee5db0a828","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"103b6a98-7079-4079-a37b-4216a77b9205","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"14d9be38-a2e8-482d-96b8-48359bed89bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d03f5fd2-ce08-4ea9-8390-2efa9b037a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:36:22.037 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.036+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a15f0 msgr2=0x7f2abc123670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:22.037 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.036+0000 7f2ac36ad640 1 --2- 172.21.15.133:0/1898745858 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a15f0 0x7f2abc123670 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:22.037 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.036+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 shutdown_connections 2024-08-16T02:36:22.037 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.036+0000 7f2ac36ad640 1 --2- 172.21.15.133:0/1898745858 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2abc0a15f0 0x7f2abc123670 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:22.037 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.036+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 >> 172.21.15.133:0/1898745858 conn(0x7f2abc0a4e90 msgr2=0x7f2abc0716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:22.037 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.036+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 shutdown_connections 2024-08-16T02:36:22.037 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:22.036+0000 7f2ac36ad640 1 -- 172.21.15.133:0/1898745858 wait complete. 2024-08-16T02:36:22.046 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-08-16T02:36:24.380 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-08-16T02:36:24.379+0000 7fba6e0a5640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-16T02:36:28.451 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-16T02:36:28.561 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.556+0000 7f5d22e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:28.562 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.560+0000 7f5d22e2d640 1 Processor -- start 2024-08-16T02:36:28.562 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.560+0000 7f5d22e2d640 1 -- start start 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.561+0000 7f5d22e2d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5d1c1361d0 0x7f5d1c1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.561+0000 7f5d22e2d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c072840 con 0x7f5d1c1332f0 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.561+0000 7f5d22e2d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f5d1c133e10 con 0x7f5d1c1361d0 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d20bc9640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f5d1c1332f0 0x7f5d1c133700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:36376/0 (socket says 172.21.15.133:36376) 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d20bc9640 1 -- 172.21.15.133:0/2188346283 learned_addr learned my addr 172.21.15.133:0/2188346283 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d1bfff640 1 --2- 172.21.15.133:0/2188346283 >> v2:172.21.15.133:3300/0 conn(0x7f5d1c1361d0 0x7f5d1c1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d1bfff640 1 -- 172.21.15.133:0/2188346283 >> v1:172.21.15.133:6789/0 conn(0x7f5d1c1332f0 legacy=0x7f5d1c133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d1bfff640 1 -- 172.21.15.133:0/2188346283 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c09ba20 con 0x7f5d1c1361d0 2024-08-16T02:36:28.564 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d1bfff640 1 --2- 172.21.15.133:0/2188346283 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1361d0 0x7f5d1c1385f0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ceb26f470e921f6 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:28.565 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d1b7fe640 1 -- 172.21.15.133:0/2188346283 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5d10009d90 con 0x7f5d1c1361d0 2024-08-16T02:36:28.565 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d1b7fe640 1 -- 172.21.15.133:0/2188346283 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5d1000e230 con 0x7f5d1c1361d0 2024-08-16T02:36:28.565 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.562+0000 7f5d1b7fe640 1 -- 172.21.15.133:0/2188346283 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5d1000e4d0 con 0x7f5d1c1361d0 2024-08-16T02:36:28.566 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.563+0000 7f5d22e2d640 1 -- 172.21.15.133:0/2188346283 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1361d0 msgr2=0x7f5d1c1385f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:28.566 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.563+0000 7f5d22e2d640 1 --2- 172.21.15.133:0/2188346283 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1361d0 0x7f5d1c1385f0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:28.566 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.563+0000 7f5d22e2d640 1 -- 172.21.15.133:0/2188346283 shutdown_connections 2024-08-16T02:36:28.566 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.563+0000 7f5d22e2d640 1 --2- 172.21.15.133:0/2188346283 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1361d0 0x7f5d1c1385f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:28.566 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.563+0000 7f5d22e2d640 1 -- 172.21.15.133:0/2188346283 >> 172.21.15.133:0/2188346283 conn(0x7f5d1c0a22e0 msgr2=0x7f5d1c0a26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:28.566 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.563+0000 7f5d22e2d640 1 -- 172.21.15.133:0/2188346283 shutdown_connections 2024-08-16T02:36:28.566 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.563+0000 7f5d22e2d640 1 -- 172.21.15.133:0/2188346283 wait complete. 2024-08-16T02:36:28.566 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.563+0000 7f5d22e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:28.567 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.565+0000 7f5d22e2d640 1 Processor -- start 2024-08-16T02:36:28.567 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.565+0000 7f5d22e2d640 1 -- start start 2024-08-16T02:36:28.568 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.566+0000 7f5d22e2d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1332f0 0x7f5d1c06f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:28.568 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.566+0000 7f5d22e2d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c1328e0 con 0x7f5d1c1332f0 2024-08-16T02:36:28.568 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.566+0000 7f5d20bc9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1332f0 0x7f5d1c06f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.566+0000 7f5d20bc9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1332f0 0x7f5d1c06f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43632/0 (socket says 172.21.15.133:43632) 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.566+0000 7f5d20bc9640 1 -- 172.21.15.133:0/4221369582 learned_addr learned my addr 172.21.15.133:0/4221369582 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.566+0000 7f5d20bc9640 1 -- 172.21.15.133:0/4221369582 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c071450 con 0x7f5d1c1332f0 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.566+0000 7f5d20bc9640 1 --2- 172.21.15.133:0/4221369582 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1332f0 0x7f5d1c06f9e0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.566+0000 7f5d197fa640 1 -- 172.21.15.133:0/4221369582 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5d0c005a60 con 0x7f5d1c1332f0 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.567+0000 7f5d197fa640 1 -- 172.21.15.133:0/4221369582 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5d0c005bc0 con 0x7f5d1c1332f0 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.567+0000 7f5d197fa640 1 -- 172.21.15.133:0/4221369582 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5d0c005e60 con 0x7f5d1c1332f0 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.567+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1c070d00 con 0x7f5d1c1332f0 2024-08-16T02:36:28.569 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.567+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d1c123e10 con 0x7f5d1c1332f0 2024-08-16T02:36:28.570 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.567+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d1c074350 con 0x7f5d1c1332f0 2024-08-16T02:36:28.570 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.568+0000 7f5d197fa640 1 -- 172.21.15.133:0/4221369582 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) ==== 49824+0+0 (crc 0 0 0) 0x7f5d0c00c7f0 con 0x7f5d1c1332f0 2024-08-16T02:36:28.570 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.568+0000 7f5d197fa640 1 --2- 172.21.15.133:0/4221369582 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5d0803dc80 0x7f5d08040190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:28.570 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.568+0000 7f5d197fa640 1 -- 172.21.15.133:0/4221369582 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2319+0+0 (crc 0 0 0) 0x7f5d0c04e310 con 0x7f5d1c1332f0 2024-08-16T02:36:28.570 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.568+0000 7f5d1bfff640 1 --2- 172.21.15.133:0/4221369582 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5d0803dc80 0x7f5d08040190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:28.570 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.569+0000 7f5d1bfff640 1 --2- 172.21.15.133:0/4221369582 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5d0803dc80 0x7f5d08040190 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:28.575 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.573+0000 7f5d197fa640 1 -- 172.21.15.133:0/4221369582 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f5d0c00dc10 con 0x7f5d1c1332f0 2024-08-16T02:36:28.768 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.766+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d1c072840 con 0x7f5d1c1332f0 2024-08-16T02:36:28.769 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.767+0000 7f5d197fa640 1 -- 172.21.15.133:0/4221369582 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6971 (crc 0 0 0) 0x7f5d1c072840 con 0x7f5d1c1332f0 2024-08-16T02:36:28.769 INFO:teuthology.misc.health.smithi133.stdout: 2024-08-16T02:36:28.769 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":11,"fsid":"7a5c695b-6aa6-4555-8e56-4a247af82439","created":"2024-08-16T02:36:18.775904+0000","modified":"2024-08-16T02:36:28.436320+0000","last_up_change":"2024-08-16T02:36:25.430618+0000","last_in_change":"2024-08-16T02:36:20.935538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:36:27.400474+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3108e258-24a8-4a0f-86a5-a3ee5db0a828","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":2249864834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6805","nonce":2249864834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":2249864834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6807","nonce":2249864834}]},"public_addr":"172.21.15.133:6804/2249864834","cluster_addr":"172.21.15.133:6805/2249864834","heartbeat_back_addr":"172.21.15.133:6802/2249864834","heartbeat_front_addr":"172.21.15.133:6807/2249864834","state":["exists","up"]},{"osd":1,"uuid":"103b6a98-7079-4079-a37b-4216a77b9205","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6803","nonce":2214080839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":2214080839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6811","nonce":2214080839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":2214080839}]},"public_addr":"172.21.15.133:6803/2214080839","cluster_addr":"172.21.15.133:6806/2214080839","heartbeat_back_addr":"172.21.15.133:6811/2214080839","heartbeat_front_addr":"172.21.15.133:6810/2214080839","state":["exists","up"]},{"osd":2,"uuid":"14d9be38-a2e8-482d-96b8-48359bed89bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":3166527287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6807","nonce":3166527287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6809","nonce":3166527287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3166527287}]},"public_addr":"172.21.15.137:6808/3166527287","cluster_addr":"172.21.15.137:6807/3166527287","heartbeat_back_addr":"172.21.15.137:6809/3166527287","heartbeat_front_addr":"172.21.15.137:6804/3166527287","state":["exists","up"]},{"osd":3,"uuid":"d03f5fd2-ce08-4ea9-8390-2efa9b037a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6805","nonce":34553420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":34553420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":34553420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":34553420}]},"public_addr":"172.21.15.137:6805/34553420","cluster_addr":"172.21.15.137:6806/34553420","heartbeat_back_addr":"172.21.15.137:6802/34553420","heartbeat_front_addr":"172.21.15.137:6803/34553420","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:36:28.772 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5d0803dc80 msgr2=0x7f5d08040190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:28.772 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 --2- 172.21.15.133:0/4221369582 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5d0803dc80 0x7f5d08040190 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:28.772 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1332f0 msgr2=0x7f5d1c06f9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:28.772 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 --2- 172.21.15.133:0/4221369582 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1332f0 0x7f5d1c06f9e0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:28.772 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 shutdown_connections 2024-08-16T02:36:28.772 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 --2- 172.21.15.133:0/4221369582 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5d0803dc80 0x7f5d08040190 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:28.772 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 --2- 172.21.15.133:0/4221369582 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d1c1332f0 0x7f5d1c06f9e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:28.773 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 >> 172.21.15.133:0/4221369582 conn(0x7f5d1c0a22e0 msgr2=0x7f5d1c0a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:28.773 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 shutdown_connections 2024-08-16T02:36:28.773 INFO:teuthology.misc.health.smithi133.stderr:2024-08-16T02:36:28.770+0000 7f5d22e2d640 1 -- 172.21.15.133:0/4221369582 wait complete. 2024-08-16T02:36:28.782 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-08-16T02:36:28.783 INFO:tasks.ceph:Creating RBD pool 2024-08-16T02:36:28.783 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-16T02:36:28.894 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.885+0000 7f50d7eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:28.895 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.894+0000 7f50d7eed640 1 Processor -- start 2024-08-16T02:36:28.895 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.894+0000 7f50d7eed640 1 -- start start 2024-08-16T02:36:28.897 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.895+0000 7f50d7eed640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f50d00a0540 0x7f50d013a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:28.897 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.895+0000 7f50d7eed640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d00a7570 con 0x7f50d00a6ea0 2024-08-16T02:36:28.897 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.895+0000 7f50d7eed640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f50d006f380 con 0x7f50d00a0540 2024-08-16T02:36:28.897 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.895+0000 7f50d5488640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f50d00a0540 0x7f50d013a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:28.897 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.895+0000 7f50d5c89640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f50d00a6ea0 0x7f50d009fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:36382/0 (socket says 172.21.15.133:36382) 2024-08-16T02:36:28.897 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.895+0000 7f50d5c89640 1 -- 172.21.15.133:0/2750777488 learned_addr learned my addr 172.21.15.133:0/2750777488 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:28.898 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.896+0000 7f50d5488640 1 -- 172.21.15.133:0/2750777488 >> v1:172.21.15.133:6789/0 conn(0x7f50d00a6ea0 legacy=0x7f50d009fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:28.898 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.896+0000 7f50d5488640 1 -- 172.21.15.133:0/2750777488 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d00a1630 con 0x7f50d00a0540 2024-08-16T02:36:28.898 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.896+0000 7f50d5488640 1 --2- 172.21.15.133:0/2750777488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d013a770 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b2eaf6fbf7c031 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:28.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.950+0000 7f50d4c87640 1 -- 172.21.15.133:0/2750777488 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f50c0009d90 con 0x7f50d00a0540 2024-08-16T02:36:28.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.950+0000 7f50d4c87640 1 -- 172.21.15.133:0/2750777488 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f50c000e230 con 0x7f50d00a0540 2024-08-16T02:36:28.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.950+0000 7f50d4c87640 1 -- 172.21.15.133:0/2750777488 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f50c000e4d0 con 0x7f50d00a0540 2024-08-16T02:36:28.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.951+0000 7f50d7eed640 1 -- 172.21.15.133:0/2750777488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 msgr2=0x7f50d013a770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:28.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.951+0000 7f50d7eed640 1 --2- 172.21.15.133:0/2750777488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d013a770 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:28.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.951+0000 7f50d7eed640 1 -- 172.21.15.133:0/2750777488 shutdown_connections 2024-08-16T02:36:28.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.951+0000 7f50d7eed640 1 --2- 172.21.15.133:0/2750777488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d013a770 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:28.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.951+0000 7f50d7eed640 1 -- 172.21.15.133:0/2750777488 >> 172.21.15.133:0/2750777488 conn(0x7f50d006ce60 msgr2=0x7f50d006d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:28.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.951+0000 7f50d7eed640 1 -- 172.21.15.133:0/2750777488 shutdown_connections 2024-08-16T02:36:28.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.951+0000 7f50d7eed640 1 -- 172.21.15.133:0/2750777488 wait complete. 2024-08-16T02:36:28.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.952+0000 7f50d7eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:28.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.953+0000 7f50d7eed640 1 Processor -- start 2024-08-16T02:36:28.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.953+0000 7f50d7eed640 1 -- start start 2024-08-16T02:36:28.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.954+0000 7f50d7eed640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d0122f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:28.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.954+0000 7f50d7eed640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01249b0 con 0x7f50d00a0540 2024-08-16T02:36:28.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.954+0000 7f50d5c89640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d0122f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:28.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.954+0000 7f50d5c89640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d0122f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43644/0 (socket says 172.21.15.133:43644) 2024-08-16T02:36:28.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.954+0000 7f50d5c89640 1 -- 172.21.15.133:0/3903796483 learned_addr learned my addr 172.21.15.133:0/3903796483 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:28.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.954+0000 7f50d5c89640 1 -- 172.21.15.133:0/3903796483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d0124170 con 0x7f50d00a0540 2024-08-16T02:36:28.958 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.954+0000 7f50d5c89640 1 --2- 172.21.15.133:0/3903796483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d0122f30 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:28.958 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.955+0000 7f50c67fc640 1 -- 172.21.15.133:0/3903796483 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f50cc005a60 con 0x7f50d00a0540 2024-08-16T02:36:28.958 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.955+0000 7f50c67fc640 1 -- 172.21.15.133:0/3903796483 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f50cc005bc0 con 0x7f50d00a0540 2024-08-16T02:36:28.958 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.955+0000 7f50c67fc640 1 -- 172.21.15.133:0/3903796483 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f50cc005e60 con 0x7f50d00a0540 2024-08-16T02:36:28.958 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.955+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d0125260 con 0x7f50d00a0540 2024-08-16T02:36:28.958 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.955+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d01259e0 con 0x7f50d00a0540 2024-08-16T02:36:28.959 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.956+0000 7f50c67fc640 1 -- 172.21.15.133:0/3903796483 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) ==== 49824+0+0 (crc 0 0 0) 0x7f50cc00c820 con 0x7f50d00a0540 2024-08-16T02:36:28.959 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.956+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d0127010 con 0x7f50d00a0540 2024-08-16T02:36:28.959 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.956+0000 7f50c67fc640 1 --2- 172.21.15.133:0/3903796483 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f50b403dc80 0x7f50b4040190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:28.959 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.956+0000 7f50c67fc640 1 -- 172.21.15.133:0/3903796483 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2319+0+0 (crc 0 0 0) 0x7f50cc04e460 con 0x7f50d00a0540 2024-08-16T02:36:28.959 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.956+0000 7f50d5488640 1 --2- 172.21.15.133:0/3903796483 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f50b403dc80 0x7f50b4040190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:28.959 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.957+0000 7f50d5488640 1 --2- 172.21.15.133:0/3903796483 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f50b403dc80 0x7f50b4040190 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:28.963 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:28.962+0000 7f50c67fc640 1 -- 172.21.15.133:0/3903796483 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f50d0127010 con 0x7f50d00a0540 2024-08-16T02:36:29.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.157+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f50d0127010 con 0x7f50d00a0540 2024-08-16T02:36:29.441 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.440+0000 7f50c67fc640 1 -- 172.21.15.133:0/3903796483 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v12) ==== 109+0+0 (crc 0 0 0) 0x7f50d0127010 con 0x7f50d00a0540 2024-08-16T02:36:29.442 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'rbd' created 2024-08-16T02:36:29.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.442+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f50b403dc80 msgr2=0x7f50b4040190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.442+0000 7f50d7eed640 1 --2- 172.21.15.133:0/3903796483 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f50b403dc80 0x7f50b4040190 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.442+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 msgr2=0x7f50d0122f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.442+0000 7f50d7eed640 1 --2- 172.21.15.133:0/3903796483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d0122f30 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.442+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 shutdown_connections 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.442+0000 7f50d7eed640 1 --2- 172.21.15.133:0/3903796483 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f50b403dc80 0x7f50b4040190 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.442+0000 7f50d7eed640 1 --2- 172.21.15.133:0/3903796483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f50d00a0540 0x7f50d0122f30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.442+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 >> 172.21.15.133:0/3903796483 conn(0x7f50d006ce60 msgr2=0x7f50d00a30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.443+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 shutdown_connections 2024-08-16T02:36:29.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.443+0000 7f50d7eed640 1 -- 172.21.15.133:0/3903796483 wait complete. 2024-08-16T02:36:29.463 DEBUG:teuthology.orchestra.run.smithi133:> rbd --cluster ceph pool init rbd 2024-08-16T02:36:29.552 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.550+0000 7f435a8e18c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:29.552 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.550+0000 7f435a8e18c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:29.553 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.551+0000 7f435a8e18c0 1 Processor -- start 2024-08-16T02:36:29.553 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.551+0000 7f435a8e18c0 1 -- start start 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.552+0000 7f435a8e18c0 1 --2- >> v2:172.21.15.133:3300/0 conn(0x25fea00 0x2694d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.552+0000 7f435a8e18c0 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x245d500 con 0x25fdee0 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.552+0000 7f435a8e18c0 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x249c130 con 0x25fea00 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.552+0000 7f43530f6640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x25fea00 0x2694d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.553+0000 7f43538f7640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x25fdee0 0x25fe2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:36396/0 (socket says 172.21.15.133:36396) 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.553+0000 7f43538f7640 1 -- 172.21.15.133:0/2455499862 learned_addr learned my addr 172.21.15.133:0/2455499862 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.553+0000 7f43530f6640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x25fea00 0x2694d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43654/0 (socket says 172.21.15.133:43654) 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.553+0000 7f43530f6640 1 -- 172.21.15.133:0/2455499862 >> v1:172.21.15.133:6789/0 conn(0x25fdee0 legacy=0x25fe2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:29.555 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.553+0000 7f43530f6640 1 -- 172.21.15.133:0/2455499862 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x24333d0 con 0x25fea00 2024-08-16T02:36:29.556 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.554+0000 7f43530f6640 1 --2- 172.21.15.133:0/2455499862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fea00 0x2694d00 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529c46a9d956c67a server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:29.556 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.554+0000 7f43528f5640 1 -- 172.21.15.133:0/2455499862 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f43400090b0 con 0x25fea00 2024-08-16T02:36:29.556 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.554+0000 7f43528f5640 1 -- 172.21.15.133:0/2455499862 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4340009210 con 0x25fea00 2024-08-16T02:36:29.556 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.554+0000 7f43528f5640 1 -- 172.21.15.133:0/2455499862 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4340007490 con 0x25fea00 2024-08-16T02:36:29.557 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.555+0000 7f435a8e18c0 1 -- 172.21.15.133:0/2455499862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fea00 msgr2=0x2694d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:29.557 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.555+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/2455499862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fea00 0x2694d00 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:29.557 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.555+0000 7f435a8e18c0 1 -- 172.21.15.133:0/2455499862 shutdown_connections 2024-08-16T02:36:29.558 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.555+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/2455499862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fea00 0x2694d00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:29.558 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.555+0000 7f435a8e18c0 1 -- 172.21.15.133:0/2455499862 >> 172.21.15.133:0/2455499862 conn(0x2446490 msgr2=0x25e26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:29.558 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.555+0000 7f435a8e18c0 1 -- 172.21.15.133:0/2455499862 shutdown_connections 2024-08-16T02:36:29.558 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.555+0000 7f435a8e18c0 1 -- 172.21.15.133:0/2455499862 wait complete. 2024-08-16T02:36:29.558 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.555+0000 7f435a8e18c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:29.558 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.556+0000 7f435a8e18c0 1 Processor -- start 2024-08-16T02:36:29.558 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.556+0000 7f435a8e18c0 1 -- start start 2024-08-16T02:36:29.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.557+0000 7f435a8e18c0 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fdee0 0x267cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:29.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.557+0000 7f435a8e18c0 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x24a6c10 con 0x25fdee0 2024-08-16T02:36:29.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f43538f7640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fdee0 0x267cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:29.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f43538f7640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fdee0 0x267cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43666/0 (socket says 172.21.15.133:43666) 2024-08-16T02:36:29.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f43538f7640 1 -- 172.21.15.133:0/178806248 learned_addr learned my addr 172.21.15.133:0/178806248 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:29.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f43538f7640 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x2441c50 con 0x25fdee0 2024-08-16T02:36:29.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f43538f7640 1 --2- 172.21.15.133:0/178806248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fdee0 0x267cc70 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:29.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f434800c490 con 0x25fdee0 2024-08-16T02:36:29.561 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f434800c5f0 con 0x25fdee0 2024-08-16T02:36:29.561 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x2433760 con 0x25fdee0 2024-08-16T02:36:29.561 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f434800c890 con 0x25fdee0 2024-08-16T02:36:29.561 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.558+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x24331c0 con 0x25fdee0 2024-08-16T02:36:29.562 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.559+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) ==== 49824+0+0 (crc 0 0 0) 0x7f434800ca90 con 0x25fdee0 2024-08-16T02:36:29.562 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.559+0000 7f433ffff640 1 --2- 172.21.15.133:0/178806248 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f433403c4c0 0x7f433403e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:29.562 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.559+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2661+0+0 (crc 0 0 0) 0x7f434804de80 con 0x25fdee0 2024-08-16T02:36:29.562 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.559+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x267f3e0 con 0x25fdee0 2024-08-16T02:36:29.562 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.560+0000 7f43530f6640 1 --2- 172.21.15.133:0/178806248 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f433403c4c0 0x7f433403e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:29.562 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:29.560+0000 7f43530f6640 1 --2- 172.21.15.133:0/178806248 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f433403c4c0 0x7f433403e9d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:30.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.233+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f434800cd90 con 0x25fdee0 2024-08-16T02:36:30.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.450+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v13) ==== 141+0+0 (crc 0 0 0) 0x7f4348007c60 con 0x25fdee0 2024-08-16T02:36:30.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.450+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x245e6a0 con 0x25fdee0 2024-08-16T02:36:30.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.455+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mon_get_version_reply(handle=1 version=13) ==== 24+0+0 (crc 0 0 0) 0x7f4348005690 con 0x25fdee0 2024-08-16T02:36:30.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.455+0000 7f43520f4640 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=13}) -- 0x7f432c002ae0 con 0x25fdee0 2024-08-16T02:36:30.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.455+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 9 ==== osd_map(13..13 src has 1..13) ==== 658+0+0 (crc 0 0 0) 0x7f4348020040 con 0x25fdee0 2024-08-16T02:36:30.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.456+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6803/2214080839 conn(0x25fea00 0x26a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:30.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.456+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 --> v2:172.21.15.133:6803/2214080839 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13) -- 0x26a2810 con 0x25fea00 2024-08-16T02:36:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.460+0000 7f4354b59640 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6803/2214080839 conn(0x25fea00 0x26a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:36:30.479 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.477+0000 7f4354b59640 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6803/2214080839 conn(0x25fea00 0x26a22d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:30.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.566+0000 7f4354b59640 1 -- 172.21.15.133:0/178806248 <== osd.1 v2:172.21.15.133:6803/2214080839 1 ==== osd_op_reply(1 rbd_trash [create] v13'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f434400a4b0 con 0x25fea00 2024-08-16T02:36:30.572 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.570+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6804/2249864834 conn(0x26f5110 0x26f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:30.572 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.570+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 --> v2:172.21.15.133:6804/2249864834 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13) -- 0x26f7bc0 con 0x26f5110 2024-08-16T02:36:30.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.572+0000 7f43538f7640 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6804/2249864834 conn(0x26f5110 0x26f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:36:30.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.576+0000 7f43538f7640 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6804/2249864834 conn(0x26f5110 0x26f75f0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:30.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.580+0000 7f43538f7640 1 -- 172.21.15.133:0/178806248 <== osd.0 v2:172.21.15.133:6804/2249864834 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f43480062a0 con 0x26f5110 2024-08-16T02:36:30.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.581+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 --> v2:172.21.15.133:6804/2249864834 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13) -- 0x26f45d0 con 0x26f5110 2024-08-16T02:36:30.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.584+0000 7f43538f7640 1 -- 172.21.15.133:0/178806248 <== osd.0 v2:172.21.15.133:6804/2249864834 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f43480062a0 con 0x26f5110 2024-08-16T02:36:30.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:30.584+0000 7f43518f3640 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f4330001f60 con 0x25fdee0 2024-08-16T02:36:31.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:31.455+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 10 ==== pool_op_reply(tid 4 (0) Success v14) ==== 55+0+0 (crc 0 0 0) 0x7f4348009d30 con 0x25fdee0 2024-08-16T02:36:31.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:31.456+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=14}) -- 0x7f4334041940 con 0x25fdee0 2024-08-16T02:36:31.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:31.456+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 11 ==== osd_map(14..14 src has 1..14) ==== 626+0+0 (crc 0 0 0) 0x7f434804f8c0 con 0x25fdee0 2024-08-16T02:36:31.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:31.456+0000 7f43520f4640 1 -- 172.21.15.133:0/178806248 --> v2:172.21.15.133:6804/2249864834 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) -- 0x7f432c005640 con 0x26f5110 2024-08-16T02:36:31.498 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:31.496+0000 7f43538f7640 1 -- 172.21.15.133:0/178806248 <== osd.0 v2:172.21.15.133:6804/2249864834 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v14'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f43480062a0 con 0x26f5110 2024-08-16T02:36:31.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:31.497+0000 7f43518f3640 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v14) -- 0x7f4330003370 con 0x25fdee0 2024-08-16T02:36:32.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.464+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 12 ==== pool_op_reply(tid 6 (0) Success v15) ==== 43+0+0 (crc 0 0 0) 0x7f4348007a30 con 0x25fdee0 2024-08-16T02:36:32.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.465+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=15}) -- 0x7f43340419f0 con 0x25fdee0 2024-08-16T02:36:32.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.465+0000 7f433ffff640 1 -- 172.21.15.133:0/178806248 <== mon.0 v2:172.21.15.133:3300/0 13 ==== osd_map(15..15 src has 1..15) ==== 654+0+0 (crc 0 0 0) 0x7f43480058d0 con 0x25fdee0 2024-08-16T02:36:32.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.465+0000 7f43520f4640 1 -- 172.21.15.133:0/178806248 --> v2:172.21.15.133:6804/2249864834 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) -- 0x7f432c0059e0 con 0x26f5110 2024-08-16T02:36:32.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.505+0000 7f43538f7640 1 -- 172.21.15.133:0/178806248 <== osd.0 v2:172.21.15.133:6804/2249864834 4 ==== osd_op_reply(7 rbd_info [write 0~19] v15'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f43480062a0 con 0x26f5110 2024-08-16T02:36:32.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.505+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6804/2249864834 conn(0x26f5110 msgr2=0x26f75f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.505+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6804/2249864834 conn(0x26f5110 0x26f75f0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6803/2214080839 conn(0x25fea00 msgr2=0x26a22d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6803/2214080839 conn(0x25fea00 0x26a22d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f433403c4c0 msgr2=0x7f433403e9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f433403c4c0 0x7f433403e9d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fdee0 msgr2=0x267cc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fdee0 0x267cc70 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 shutdown_connections 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6804/2249864834 conn(0x26f5110 0x26f75f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> v2:172.21.15.133:6803/2214080839 conn(0x25fea00 0x26a22d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f433403c4c0 0x7f433403e9d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 --2- 172.21.15.133:0/178806248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x25fdee0 0x267cc70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 >> 172.21.15.133:0/178806248 conn(0x2446490 msgr2=0x2691fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 shutdown_connections 2024-08-16T02:36:32.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.506+0000 7f435a8e18c0 1 -- 172.21.15.133:0/178806248 wait complete. 2024-08-16T02:36:32.513 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-16T02:36:32.513 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-08-16T02:36:32.514 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-16T02:36:32.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.621+0000 7f09216ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:32.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.626+0000 7f09216ad640 1 Processor -- start 2024-08-16T02:36:32.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.626+0000 7f09216ad640 1 -- start start 2024-08-16T02:36:32.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.627+0000 7f09216ad640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f091c135490 0x7f091c1358a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:32.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.627+0000 7f09216ad640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f091c0a4f10 con 0x7f091c071390 2024-08-16T02:36:32.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.627+0000 7f09216ad640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f091c0a5140 con 0x7f091c135490 2024-08-16T02:36:32.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.627+0000 7f091affd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f091c135490 0x7f091c1358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:32.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.627+0000 7f0913fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f091c071390 0x7f091c0717a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34228/0 (socket says 172.21.15.133:34228) 2024-08-16T02:36:32.631 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.627+0000 7f091affd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f091c135490 0x7f091c1358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55876/0 (socket says 172.21.15.133:55876) 2024-08-16T02:36:32.631 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.627+0000 7f0913fff640 1 -- 172.21.15.133:0/2282265990 learned_addr learned my addr 172.21.15.133:0/2282265990 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:32.631 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.628+0000 7f091affd640 1 -- 172.21.15.133:0/2282265990 >> v1:172.21.15.133:6789/0 conn(0x7f091c071390 legacy=0x7f091c0717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.631 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.628+0000 7f091affd640 1 -- 172.21.15.133:0/2282265990 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091c072b20 con 0x7f091c135490 2024-08-16T02:36:32.631 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.628+0000 7f091affd640 1 --2- 172.21.15.133:0/2282265990 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c135490 0x7f091c1358a0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada78fa070f16536 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:32.631 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.628+0000 7f091a7fc640 1 -- 172.21.15.133:0/2282265990 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f090c009d90 con 0x7f091c135490 2024-08-16T02:36:32.631 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.628+0000 7f091a7fc640 1 -- 172.21.15.133:0/2282265990 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f090c00e230 con 0x7f091c135490 2024-08-16T02:36:32.631 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.628+0000 7f091a7fc640 1 -- 172.21.15.133:0/2282265990 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f090c00e4d0 con 0x7f091c135490 2024-08-16T02:36:32.632 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.629+0000 7f09216ad640 1 -- 172.21.15.133:0/2282265990 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c135490 msgr2=0x7f091c1358a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.632 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.629+0000 7f09216ad640 1 --2- 172.21.15.133:0/2282265990 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c135490 0x7f091c1358a0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.632 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.629+0000 7f09216ad640 1 -- 172.21.15.133:0/2282265990 shutdown_connections 2024-08-16T02:36:32.632 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.629+0000 7f09216ad640 1 --2- 172.21.15.133:0/2282265990 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c135490 0x7f091c1358a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.633 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.629+0000 7f09216ad640 1 -- 172.21.15.133:0/2282265990 >> 172.21.15.133:0/2282265990 conn(0x7f091c0a5880 msgr2=0x7f091c0a5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:32.633 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.629+0000 7f09216ad640 1 -- 172.21.15.133:0/2282265990 shutdown_connections 2024-08-16T02:36:32.633 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.629+0000 7f09216ad640 1 -- 172.21.15.133:0/2282265990 wait complete. 2024-08-16T02:36:32.633 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.629+0000 7f09216ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:32.633 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.630+0000 7f09216ad640 1 Processor -- start 2024-08-16T02:36:32.633 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.630+0000 7f09216ad640 1 -- start start 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.631+0000 7f09216ad640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c071390 0x7f091c1244c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.631+0000 7f09216ad640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1267d0 con 0x7f091c071390 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f091affd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c071390 0x7f091c1244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f091affd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c071390 0x7f091c1244c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55886/0 (socket says 172.21.15.133:55886) 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f091affd640 1 -- 172.21.15.133:0/804526090 learned_addr learned my addr 172.21.15.133:0/804526090 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f091affd640 1 -- 172.21.15.133:0/804526090 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091c128510 con 0x7f091c071390 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f091affd640 1 --2- 172.21.15.133:0/804526090 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c071390 0x7f091c1244c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f09137fe640 1 -- 172.21.15.133:0/804526090 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f090c014620 con 0x7f091c071390 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f09137fe640 1 -- 172.21.15.133:0/804526090 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f090c014780 con 0x7f091c071390 2024-08-16T02:36:32.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f091c127880 con 0x7f091c071390 2024-08-16T02:36:32.636 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f09137fe640 1 -- 172.21.15.133:0/804526090 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f090c014a20 con 0x7f091c071390 2024-08-16T02:36:32.636 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.632+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f091c125f40 con 0x7f091c071390 2024-08-16T02:36:32.636 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.633+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f091c0008d0 con 0x7f091c071390 2024-08-16T02:36:32.636 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.633+0000 7f09137fe640 1 -- 172.21.15.133:0/804526090 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f090c014bd0 con 0x7f091c071390 2024-08-16T02:36:32.636 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.633+0000 7f09137fe640 1 --2- 172.21.15.133:0/804526090 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f08fc03dd10 0x7f08fc040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:32.637 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.634+0000 7f09137fe640 1 -- 172.21.15.133:0/804526090 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f090c019040 con 0x7f091c071390 2024-08-16T02:36:32.637 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.634+0000 7f0913fff640 1 --2- 172.21.15.133:0/804526090 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f08fc03dd10 0x7f08fc040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:32.637 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.634+0000 7f0913fff640 1 --2- 172.21.15.133:0/804526090 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f08fc03dd10 0x7f08fc040220 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:32.640 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.639+0000 7f09137fe640 1 -- 172.21.15.133:0/804526090 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f090c018970 con 0x7f091c071390 2024-08-16T02:36:32.826 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.824+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f091c127540 con 0x7f091c071390 2024-08-16T02:36:32.826 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.824+0000 7f09137fe640 1 -- 172.21.15.133:0/804526090 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v1) ==== 86+0+61 (crc 0 0 0) 0x7f090c018190 con 0x7f091c071390 2024-08-16T02:36:32.826 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:32.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f08fc03dd10 msgr2=0x7f08fc040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 --2- 172.21.15.133:0/804526090 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f08fc03dd10 0x7f08fc040220 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c071390 msgr2=0x7f091c1244c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 --2- 172.21.15.133:0/804526090 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c071390 0x7f091c1244c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 shutdown_connections 2024-08-16T02:36:32.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 --2- 172.21.15.133:0/804526090 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f08fc03dd10 0x7f08fc040220 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 --2- 172.21.15.133:0/804526090 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f091c071390 0x7f091c1244c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 >> 172.21.15.133:0/804526090 conn(0x7f091c0a5880 msgr2=0x7f091c06fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:32.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 shutdown_connections 2024-08-16T02:36:32.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.827+0000 7f09216ad640 1 -- 172.21.15.133:0/804526090 wait complete. 2024-08-16T02:36:32.839 INFO:teuthology.orchestra.run.smithi133.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-08-16T02:36:32.839 INFO:tasks.ceph_manager:config epoch is 1 2024-08-16T02:36:32.839 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-16T02:36:32.839 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-16T02:36:32.840 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-16T02:36:32.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.943+0000 7f711c392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:32.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.947+0000 7f711c392640 1 Processor -- start 2024-08-16T02:36:32.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.947+0000 7f711c392640 1 -- start start 2024-08-16T02:36:32.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711c392640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f711409c540 0x7f711413a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:32.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711c392640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711409eec0 con 0x7f711409ba20 2024-08-16T02:36:32.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711c392640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f711406e8b0 con 0x7f711409c540 2024-08-16T02:36:32.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711992d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f711409c540 0x7f711413a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:32.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711a12e640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f711409ba20 0x7f711409be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34238/0 (socket says 172.21.15.133:34238) 2024-08-16T02:36:32.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711a12e640 1 -- 172.21.15.133:0/2816427555 learned_addr learned my addr 172.21.15.133:0/2816427555 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:32.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711992d640 1 -- 172.21.15.133:0/2816427555 >> v1:172.21.15.133:6789/0 conn(0x7f711409ba20 legacy=0x7f711409be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711912c640 1 -- 172.21.15.133:0/2816427555 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346792505 0 0) 0x7f711409eec0 con 0x7f711409ba20 2024-08-16T02:36:32.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.948+0000 7f711992d640 1 -- 172.21.15.133:0/2816427555 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711406ee60 con 0x7f711409c540 2024-08-16T02:36:32.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.949+0000 7f711992d640 1 --2- 172.21.15.133:0/2816427555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409c540 0x7f711413a770 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93023a74a290e583 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:32.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.949+0000 7f711912c640 1 -- 172.21.15.133:0/2816427555 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7104009d90 con 0x7f711409c540 2024-08-16T02:36:32.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.949+0000 7f711912c640 1 -- 172.21.15.133:0/2816427555 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f710400e230 con 0x7f711409c540 2024-08-16T02:36:32.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.949+0000 7f711912c640 1 -- 172.21.15.133:0/2816427555 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f710400e4d0 con 0x7f711409c540 2024-08-16T02:36:32.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.949+0000 7f711c392640 1 -- 172.21.15.133:0/2816427555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409c540 msgr2=0x7f711413a770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:32.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.949+0000 7f711c392640 1 --2- 172.21.15.133:0/2816427555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409c540 0x7f711413a770 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.950+0000 7f711c392640 1 -- 172.21.15.133:0/2816427555 shutdown_connections 2024-08-16T02:36:32.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.950+0000 7f711c392640 1 --2- 172.21.15.133:0/2816427555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409c540 0x7f711413a770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:32.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.950+0000 7f711c392640 1 -- 172.21.15.133:0/2816427555 >> 172.21.15.133:0/2816427555 conn(0x7f711409f1d0 msgr2=0x7f711409f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:32.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.950+0000 7f711c392640 1 -- 172.21.15.133:0/2816427555 shutdown_connections 2024-08-16T02:36:32.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.950+0000 7f711c392640 1 -- 172.21.15.133:0/2816427555 wait complete. 2024-08-16T02:36:32.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.950+0000 7f711c392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:32.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.951+0000 7f711c392640 1 Processor -- start 2024-08-16T02:36:32.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.951+0000 7f711c392640 1 -- start start 2024-08-16T02:36:32.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.952+0000 7f711c392640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409ba20 0x7f7114123660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:32.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.952+0000 7f711c392640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f7114123210 con 0x7f711409ba20 2024-08-16T02:36:32.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.952+0000 7f711a12e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409ba20 0x7f7114123660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:32.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.952+0000 7f711a12e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409ba20 0x7f7114123660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55900/0 (socket says 172.21.15.133:55900) 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.952+0000 7f711a12e640 1 -- 172.21.15.133:0/3438363412 learned_addr learned my addr 172.21.15.133:0/3438363412 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.952+0000 7f711a12e640 1 -- 172.21.15.133:0/3438363412 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7114124770 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.953+0000 7f711a12e640 1 --2- 172.21.15.133:0/3438363412 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409ba20 0x7f7114123660 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.953+0000 7f710affd640 1 -- 172.21.15.133:0/3438363412 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7110004b80 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.953+0000 7f710affd640 1 -- 172.21.15.133:0/3438363412 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f7110004ce0 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.953+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7114122e70 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.953+0000 7f710affd640 1 -- 172.21.15.133:0/3438363412 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7110007120 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.953+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7114126010 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.954+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711409be30 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.954+0000 7f710affd640 1 -- 172.21.15.133:0/3438363412 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f71100072d0 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.954+0000 7f710affd640 1 --2- 172.21.15.133:0/3438363412 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f70f803dd40 0x7f70f8040250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.954+0000 7f710affd640 1 -- 172.21.15.133:0/3438363412 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f711004e880 con 0x7f711409ba20 2024-08-16T02:36:32.957 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.954+0000 7f711992d640 1 --2- 172.21.15.133:0/3438363412 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f70f803dd40 0x7f70f8040250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:32.958 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.955+0000 7f711992d640 1 --2- 172.21.15.133:0/3438363412 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f70f803dd40 0x7f70f8040250 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:32.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:32.960+0000 7f710affd640 1 -- 172.21.15.133:0/3438363412 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f711000adb0 con 0x7f711409ba20 2024-08-16T02:36:33.192 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.191+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f711406bf10 con 0x7f711409ba20 2024-08-16T02:36:33.194 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.193+0000 7f710affd640 1 -- 172.21.15.133:0/3438363412 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+96286 (crc 0 0 0) 0x7f711000d150 con 0x7f711409ba20 2024-08-16T02:36:33.196 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:33.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f70f803dd40 msgr2=0x7f70f8040250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 --2- 172.21.15.133:0/3438363412 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f70f803dd40 0x7f70f8040250 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409ba20 msgr2=0x7f7114123660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 --2- 172.21.15.133:0/3438363412 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409ba20 0x7f7114123660 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 shutdown_connections 2024-08-16T02:36:33.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 --2- 172.21.15.133:0/3438363412 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f70f803dd40 0x7f70f8040250 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 --2- 172.21.15.133:0/3438363412 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f711409ba20 0x7f7114123660 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 >> 172.21.15.133:0/3438363412 conn(0x7f711409f1d0 msgr2=0x7f71140712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.198 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 shutdown_connections 2024-08-16T02:36:33.198 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.195+0000 7f711c392640 1 -- 172.21.15.133:0/3438363412 wait complete. 2024-08-16T02:36:33.207 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":545649490},{"type":"v1","addr":"172.21.15.137:6801","nonce":545649490}]},"active_addr":"172.21.15.137:6801/545649490","active_change":"2024-08-16T02:36:23.365657+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1580683006}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1756691377}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1891106432}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":2847886140}]}]} 2024-08-16T02:36:33.209 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-16T02:36:33.209 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-16T02:36:33.209 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:36:33.317 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.312+0000 7f575706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.318 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.317+0000 7f575706d640 1 Processor -- start 2024-08-16T02:36:33.318 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.317+0000 7f575706d640 1 -- start start 2024-08-16T02:36:33.319 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.318+0000 7f575706d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f575009e340 0x7f5750138540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.318+0000 7f575706d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57500a0f70 con 0x7f575006f660 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.318+0000 7f575706d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f57500a0d00 con 0x7f575009e340 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.318+0000 7f5754e09640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f575006f660 0x7f575009dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34252/0 (socket says 172.21.15.133:34252) 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.318+0000 7f5754e09640 1 -- 172.21.15.133:0/3348924332 learned_addr learned my addr 172.21.15.133:0/3348924332 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.318+0000 7f574ffff640 1 --2- 172.21.15.133:0/3348924332 >> v2:172.21.15.133:3300/0 conn(0x7f575009e340 0x7f5750138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.319+0000 7f574f7fe640 1 -- 172.21.15.133:0/3348924332 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894641650 0 0) 0x7f57500a0f70 con 0x7f575006f660 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.319+0000 7f574f7fe640 1 -- 172.21.15.133:0/3348924332 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5740003580 con 0x7f575006f660 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.319+0000 7f574ffff640 1 -- 172.21.15.133:0/3348924332 >> v1:172.21.15.133:6789/0 conn(0x7f575006f660 legacy=0x7f575009dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.319+0000 7f574ffff640 1 -- 172.21.15.133:0/3348924332 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575009ee50 con 0x7f575009e340 2024-08-16T02:36:33.320 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.319+0000 7f574ffff640 1 --2- 172.21.15.133:0/3348924332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575009e340 0x7f5750138540 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc3ec3468835ed server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.319+0000 7f574f7fe640 1 -- 172.21.15.133:0/3348924332 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f573c009d90 con 0x7f575009e340 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.319+0000 7f574f7fe640 1 -- 172.21.15.133:0/3348924332 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f573c00e230 con 0x7f575009e340 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.319+0000 7f574f7fe640 1 -- 172.21.15.133:0/3348924332 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f573c00e4d0 con 0x7f575009e340 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.320+0000 7f575706d640 1 -- 172.21.15.133:0/3348924332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575009e340 msgr2=0x7f5750138540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.320+0000 7f575706d640 1 --2- 172.21.15.133:0/3348924332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575009e340 0x7f5750138540 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.320+0000 7f575706d640 1 -- 172.21.15.133:0/3348924332 shutdown_connections 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.320+0000 7f575706d640 1 --2- 172.21.15.133:0/3348924332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575009e340 0x7f5750138540 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.320+0000 7f575706d640 1 -- 172.21.15.133:0/3348924332 >> 172.21.15.133:0/3348924332 conn(0x7f575009f1d0 msgr2=0x7f575009f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.320+0000 7f575706d640 1 -- 172.21.15.133:0/3348924332 shutdown_connections 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.320+0000 7f575706d640 1 -- 172.21.15.133:0/3348924332 wait complete. 2024-08-16T02:36:33.322 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.320+0000 7f575706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.323 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.322+0000 7f575706d640 1 Processor -- start 2024-08-16T02:36:33.323 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.322+0000 7f575706d640 1 -- start start 2024-08-16T02:36:33.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.323+0000 7f575706d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575006f660 0x7f5750121040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.323+0000 7f575706d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f5750123120 con 0x7f575006f660 2024-08-16T02:36:33.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.323+0000 7f5754e09640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575006f660 0x7f5750121040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.323+0000 7f5754e09640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575006f660 0x7f5750121040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55920/0 (socket says 172.21.15.133:55920) 2024-08-16T02:36:33.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.323+0000 7f5754e09640 1 -- 172.21.15.133:0/4005540480 learned_addr learned my addr 172.21.15.133:0/4005540480 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:33.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.323+0000 7f5754e09640 1 -- 172.21.15.133:0/4005540480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5750120990 con 0x7f575006f660 2024-08-16T02:36:33.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.324+0000 7f5754e09640 1 --2- 172.21.15.133:0/4005540480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575006f660 0x7f5750121040 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.324+0000 7f574d7fa640 1 -- 172.21.15.133:0/4005540480 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5744005a60 con 0x7f575006f660 2024-08-16T02:36:33.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.324+0000 7f574d7fa640 1 -- 172.21.15.133:0/4005540480 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5744005bc0 con 0x7f575006f660 2024-08-16T02:36:33.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.324+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5750120ca0 con 0x7f575006f660 2024-08-16T02:36:33.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.324+0000 7f574d7fa640 1 -- 172.21.15.133:0/4005540480 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5744005e60 con 0x7f575006f660 2024-08-16T02:36:33.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.324+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5750123c60 con 0x7f575006f660 2024-08-16T02:36:33.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.325+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5750122f70 con 0x7f575006f660 2024-08-16T02:36:33.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.325+0000 7f574d7fa640 1 -- 172.21.15.133:0/4005540480 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f57440070b0 con 0x7f575006f660 2024-08-16T02:36:33.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.325+0000 7f574d7fa640 1 --2- 172.21.15.133:0/4005540480 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f573403dd10 0x7f5734040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.325+0000 7f574d7fa640 1 -- 172.21.15.133:0/4005540480 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f574404e710 con 0x7f575006f660 2024-08-16T02:36:33.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.326+0000 7f574ffff640 1 --2- 172.21.15.133:0/4005540480 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f573403dd10 0x7f5734040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.328 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.326+0000 7f574ffff640 1 --2- 172.21.15.133:0/4005540480 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f573403dd10 0x7f5734040220 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.331 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.330+0000 7f574d7fa640 1 -- 172.21.15.133:0/4005540480 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f574400d840 con 0x7f575006f660 2024-08-16T02:36:33.519 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.517+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f575009dc30 con 0x7f575006f660 2024-08-16T02:36:33.519 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.518+0000 7f574d7fa640 1 -- 172.21.15.133:0/4005540480 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8883 (crc 0 0 0) 0x7f574400d660 con 0x7f575006f660 2024-08-16T02:36:33.519 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:33.519 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":15,"fsid":"7a5c695b-6aa6-4555-8e56-4a247af82439","created":"2024-08-16T02:36:18.775904+0000","modified":"2024-08-16T02:36:32.458847+0000","last_up_change":"2024-08-16T02:36:25.430618+0000","last_in_change":"2024-08-16T02:36:20.935538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:36:27.400474+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:36:29.162647+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3108e258-24a8-4a0f-86a5-a3ee5db0a828","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":2249864834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6805","nonce":2249864834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":2249864834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6807","nonce":2249864834}]},"public_addr":"172.21.15.133:6804/2249864834","cluster_addr":"172.21.15.133:6805/2249864834","heartbeat_back_addr":"172.21.15.133:6802/2249864834","heartbeat_front_addr":"172.21.15.133:6807/2249864834","state":["exists","up"]},{"osd":1,"uuid":"103b6a98-7079-4079-a37b-4216a77b9205","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6803","nonce":2214080839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":2214080839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6811","nonce":2214080839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":2214080839}]},"public_addr":"172.21.15.133:6803/2214080839","cluster_addr":"172.21.15.133:6806/2214080839","heartbeat_back_addr":"172.21.15.133:6811/2214080839","heartbeat_front_addr":"172.21.15.133:6810/2214080839","state":["exists","up"]},{"osd":2,"uuid":"14d9be38-a2e8-482d-96b8-48359bed89bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":3166527287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6807","nonce":3166527287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6809","nonce":3166527287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3166527287}]},"public_addr":"172.21.15.137:6808/3166527287","cluster_addr":"172.21.15.137:6807/3166527287","heartbeat_back_addr":"172.21.15.137:6809/3166527287","heartbeat_front_addr":"172.21.15.137:6804/3166527287","state":["exists","up"]},{"osd":3,"uuid":"d03f5fd2-ce08-4ea9-8390-2efa9b037a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6805","nonce":34553420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":34553420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":34553420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":34553420}]},"public_addr":"172.21.15.137:6805/34553420","cluster_addr":"172.21.15.137:6806/34553420","heartbeat_back_addr":"172.21.15.137:6802/34553420","heartbeat_front_addr":"172.21.15.137:6803/34553420","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:36:33.521 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f573403dd10 msgr2=0x7f5734040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.521 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 --2- 172.21.15.133:0/4005540480 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f573403dd10 0x7f5734040220 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.522 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575006f660 msgr2=0x7f5750121040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.522 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 --2- 172.21.15.133:0/4005540480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575006f660 0x7f5750121040 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.522 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 shutdown_connections 2024-08-16T02:36:33.522 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 --2- 172.21.15.133:0/4005540480 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f573403dd10 0x7f5734040220 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.522 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 --2- 172.21.15.133:0/4005540480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f575006f660 0x7f5750121040 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.522 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 >> 172.21.15.133:0/4005540480 conn(0x7f575009f1d0 msgr2=0x7f57500725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.522 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 shutdown_connections 2024-08-16T02:36:33.522 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.520+0000 7f575706d640 1 -- 172.21.15.133:0/4005540480 wait complete. 2024-08-16T02:36:33.532 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-16T02:36:33.532 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:36:33.641 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.636+0000 7faf12592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.642 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.641+0000 7faf12592640 1 Processor -- start 2024-08-16T02:36:33.642 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.641+0000 7faf12592640 1 -- start start 2024-08-16T02:36:33.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf12592640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7faf0c138490 0x7faf0c13a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf12592640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf0c0a4f10 con 0x7faf0c135490 2024-08-16T02:36:33.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf12592640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7faf0c135fb0 con 0x7faf0c138490 2024-08-16T02:36:33.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0bfff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7faf0c135490 0x7faf0c1358a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34268/0 (socket says 172.21.15.133:34268) 2024-08-16T02:36:33.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0bfff640 1 -- 172.21.15.133:0/2532245518 learned_addr learned my addr 172.21.15.133:0/2532245518 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:33.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0b7fe640 1 --2- 172.21.15.133:0/2532245518 >> v2:172.21.15.133:3300/0 conn(0x7faf0c138490 0x7faf0c13a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.645 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0affd640 1 -- 172.21.15.133:0/2532245518 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1210441448 0 0) 0x7faf0c0a4f10 con 0x7faf0c135490 2024-08-16T02:36:33.645 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0affd640 1 -- 172.21.15.133:0/2532245518 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faef4003580 con 0x7faf0c135490 2024-08-16T02:36:33.645 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0b7fe640 1 -- 172.21.15.133:0/2532245518 >> v1:172.21.15.133:6789/0 conn(0x7faf0c135490 legacy=0x7faf0c1358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.645 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0b7fe640 1 -- 172.21.15.133:0/2532245518 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c072b20 con 0x7faf0c138490 2024-08-16T02:36:33.645 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0b7fe640 1 --2- 172.21.15.133:0/2532245518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c138490 0x7faf0c13a8b0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a571f365227fc117 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.645 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0affd640 1 -- 172.21.15.133:0/2532245518 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faf00009d90 con 0x7faf0c138490 2024-08-16T02:36:33.645 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0affd640 1 -- 172.21.15.133:0/2532245518 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7faf0000e230 con 0x7faf0c138490 2024-08-16T02:36:33.645 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.643+0000 7faf0affd640 1 -- 172.21.15.133:0/2532245518 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faf0000e4d0 con 0x7faf0c138490 2024-08-16T02:36:33.646 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.644+0000 7faf12592640 1 -- 172.21.15.133:0/2532245518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c138490 msgr2=0x7faf0c13a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.646 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.644+0000 7faf12592640 1 --2- 172.21.15.133:0/2532245518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c138490 0x7faf0c13a8b0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.646 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.644+0000 7faf12592640 1 -- 172.21.15.133:0/2532245518 shutdown_connections 2024-08-16T02:36:33.647 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.644+0000 7faf12592640 1 --2- 172.21.15.133:0/2532245518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c138490 0x7faf0c13a8b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.647 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.644+0000 7faf12592640 1 -- 172.21.15.133:0/2532245518 >> 172.21.15.133:0/2532245518 conn(0x7faf0c0a5880 msgr2=0x7faf0c0a5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.647 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.644+0000 7faf12592640 1 -- 172.21.15.133:0/2532245518 shutdown_connections 2024-08-16T02:36:33.647 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.644+0000 7faf12592640 1 -- 172.21.15.133:0/2532245518 wait complete. 2024-08-16T02:36:33.647 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.644+0000 7faf12592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.647 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.646+0000 7faf12592640 1 Processor -- start 2024-08-16T02:36:33.647 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.646+0000 7faf12592640 1 -- start start 2024-08-16T02:36:33.648 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf12592640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c135490 0x7faf0c1c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf12592640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c120c50 con 0x7faf0c135490 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf0bfff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c135490 0x7faf0c1c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf0bfff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c135490 0x7faf0c1c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55944/0 (socket says 172.21.15.133:55944) 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf0bfff640 1 -- 172.21.15.133:0/2038532333 learned_addr learned my addr 172.21.15.133:0/2038532333 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf0bfff640 1 -- 172.21.15.133:0/2038532333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c123000 con 0x7faf0c135490 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf0bfff640 1 --2- 172.21.15.133:0/2038532333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c135490 0x7faf0c1c9d50 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf08ff9640 1 -- 172.21.15.133:0/2038532333 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faefc005a60 con 0x7faf0c135490 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf08ff9640 1 -- 172.21.15.133:0/2038532333 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7faefc005bc0 con 0x7faf0c135490 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0c122650 con 0x7faf0c135490 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.647+0000 7faf08ff9640 1 -- 172.21.15.133:0/2038532333 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faefc005e60 con 0x7faf0c135490 2024-08-16T02:36:33.649 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.648+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0c123ff0 con 0x7faf0c135490 2024-08-16T02:36:33.650 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.648+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf0c1358a0 con 0x7faf0c135490 2024-08-16T02:36:33.650 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.649+0000 7faf08ff9640 1 -- 172.21.15.133:0/2038532333 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7faefc0070b0 con 0x7faf0c135490 2024-08-16T02:36:33.650 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.649+0000 7faf08ff9640 1 --2- 172.21.15.133:0/2038532333 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7faef803dcc0 0x7faef80401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.650 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.649+0000 7faf08ff9640 1 -- 172.21.15.133:0/2038532333 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7faefc04e750 con 0x7faf0c135490 2024-08-16T02:36:33.651 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.649+0000 7faf0b7fe640 1 --2- 172.21.15.133:0/2038532333 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7faef803dcc0 0x7faef80401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.651 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.650+0000 7faf0b7fe640 1 --2- 172.21.15.133:0/2038532333 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7faef803dcc0 0x7faef80401d0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.654 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.654+0000 7faf08ff9640 1 -- 172.21.15.133:0/2038532333 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7faefc00dc60 con 0x7faf0c135490 2024-08-16T02:36:33.841 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.840+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf0c073000 con 0x7faf0c135490 2024-08-16T02:36:33.841 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.841+0000 7faf08ff9640 1 -- 172.21.15.133:0/2038532333 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8883 (crc 0 0 0) 0x7faefc00d660 con 0x7faf0c135490 2024-08-16T02:36:33.841 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:33.842 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":15,"fsid":"7a5c695b-6aa6-4555-8e56-4a247af82439","created":"2024-08-16T02:36:18.775904+0000","modified":"2024-08-16T02:36:32.458847+0000","last_up_change":"2024-08-16T02:36:25.430618+0000","last_in_change":"2024-08-16T02:36:20.935538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:36:27.400474+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:36:29.162647+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3108e258-24a8-4a0f-86a5-a3ee5db0a828","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":2249864834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6805","nonce":2249864834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":2249864834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6807","nonce":2249864834}]},"public_addr":"172.21.15.133:6804/2249864834","cluster_addr":"172.21.15.133:6805/2249864834","heartbeat_back_addr":"172.21.15.133:6802/2249864834","heartbeat_front_addr":"172.21.15.133:6807/2249864834","state":["exists","up"]},{"osd":1,"uuid":"103b6a98-7079-4079-a37b-4216a77b9205","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6803","nonce":2214080839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":2214080839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6811","nonce":2214080839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":2214080839}]},"public_addr":"172.21.15.133:6803/2214080839","cluster_addr":"172.21.15.133:6806/2214080839","heartbeat_back_addr":"172.21.15.133:6811/2214080839","heartbeat_front_addr":"172.21.15.133:6810/2214080839","state":["exists","up"]},{"osd":2,"uuid":"14d9be38-a2e8-482d-96b8-48359bed89bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":3166527287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6807","nonce":3166527287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6809","nonce":3166527287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3166527287}]},"public_addr":"172.21.15.137:6808/3166527287","cluster_addr":"172.21.15.137:6807/3166527287","heartbeat_back_addr":"172.21.15.137:6809/3166527287","heartbeat_front_addr":"172.21.15.137:6804/3166527287","state":["exists","up"]},{"osd":3,"uuid":"d03f5fd2-ce08-4ea9-8390-2efa9b037a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6805","nonce":34553420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":34553420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":34553420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":34553420}]},"public_addr":"172.21.15.137:6805/34553420","cluster_addr":"172.21.15.137:6806/34553420","heartbeat_back_addr":"172.21.15.137:6802/34553420","heartbeat_front_addr":"172.21.15.137:6803/34553420","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7faef803dcc0 msgr2=0x7faef80401d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 --2- 172.21.15.133:0/2038532333 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7faef803dcc0 0x7faef80401d0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c135490 msgr2=0x7faf0c1c9d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 --2- 172.21.15.133:0/2038532333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c135490 0x7faf0c1c9d50 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 shutdown_connections 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 --2- 172.21.15.133:0/2038532333 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7faef803dcc0 0x7faef80401d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 --2- 172.21.15.133:0/2038532333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf0c135490 0x7faf0c1c9d50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 >> 172.21.15.133:0/2038532333 conn(0x7faf0c0a5880 msgr2=0x7faf0c070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 shutdown_connections 2024-08-16T02:36:33.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.843+0000 7faf12592640 1 -- 172.21.15.133:0/2038532333 wait complete. 2024-08-16T02:36:33.855 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-16T02:36:33.856 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-16T02:36:33.856 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-16T02:36:33.857 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-16T02:36:33.978 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.972+0000 7fc87f0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.978 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.972+0000 7efcf4752640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.978 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.972+0000 7fece6112640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.972+0000 7ffba9b52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.978+0000 7efcf4752640 1 Processor -- start 2024-08-16T02:36:33.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.978+0000 7efcf4752640 1 -- start start 2024-08-16T02:36:33.980 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.978+0000 7fc87f0ed640 1 Processor -- start 2024-08-16T02:36:33.980 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.978+0000 7fc87f0ed640 1 -- start start 2024-08-16T02:36:33.980 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.978+0000 7fece6112640 1 Processor -- start 2024-08-16T02:36:33.980 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.978+0000 7fece6112640 1 -- start start 2024-08-16T02:36:33.981 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.978+0000 7ffba9b52640 1 Processor -- start 2024-08-16T02:36:33.981 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.978+0000 7ffba9b52640 1 -- start start 2024-08-16T02:36:33.981 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fc87f0ed640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fc878138490 0x7fc87813a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.981 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fc87f0ed640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8780a4f10 con 0x7fc878135490 2024-08-16T02:36:33.981 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fc87f0ed640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7fc878135fb0 con 0x7fc878138490 2024-08-16T02:36:33.982 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7efcf4752640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7efcec09c540 0x7efcec13a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.982 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7efcf4752640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcec06f900 con 0x7efcec09ba20 2024-08-16T02:36:33.982 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7efcf4752640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7efcec09fff0 con 0x7efcec09c540 2024-08-16T02:36:33.983 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fc87ce89640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fc878135490 0x7fc8781358a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34276/0 (socket says 172.21.15.133:34276) 2024-08-16T02:36:33.983 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fc87ce89640 1 -- 172.21.15.133:0/2188267116 learned_addr learned my addr 172.21.15.133:0/2188267116 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:33.983 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fc877fff640 1 --2- 172.21.15.133:0/2188267116 >> v2:172.21.15.133:3300/0 conn(0x7fc878138490 0x7fc87813a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.983 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7efcf24ee640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7efcec09ba20 0x7efcec09be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34278/0 (socket says 172.21.15.133:34278) 2024-08-16T02:36:33.984 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7efcf24ee640 1 -- 172.21.15.133:0/3779039645 learned_addr learned my addr 172.21.15.133:0/3779039645 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:33.984 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fece6112640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fece009aba0 0x7fece0136c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.984 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fece6112640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece009dea0 con 0x7fece009a080 2024-08-16T02:36:33.984 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7efcf1ced640 1 --2- 172.21.15.133:0/3779039645 >> v2:172.21.15.133:3300/0 conn(0x7efcec09c540 0x7efcec13a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fecdf7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fece009a080 0x7fece009a490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34290/0 (socket says 172.21.15.133:34290) 2024-08-16T02:36:33.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.979+0000 7fecdf7fe640 1 -- 172.21.15.133:0/3793712701 learned_addr learned my addr 172.21.15.133:0/3793712701 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:33.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fece6112640 1 -- 172.21.15.133:0/3793712701 --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7fece009e260 con 0x7fece009aba0 2024-08-16T02:36:33.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fecdeffd640 1 --2- 172.21.15.133:0/3793712701 >> v2:172.21.15.133:3300/0 conn(0x7fece009aba0 0x7fece0136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fc877fff640 1 -- 172.21.15.133:0/2188267116 >> v1:172.21.15.133:6789/0 conn(0x7fc878135490 legacy=0x7fc8781358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fc877fff640 1 -- 172.21.15.133:0/2188267116 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc878072b20 con 0x7fc878138490 2024-08-16T02:36:33.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7ffba9b52640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7ffba4136270 0x7ffba4138690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:33.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7efcf14ec640 1 -- 172.21.15.133:0/3779039645 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862837083 0 0) 0x7efcec06f900 con 0x7efcec09ba20 2024-08-16T02:36:33.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7ffba9b52640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffba4133eb0 con 0x7ffba4133390 2024-08-16T02:36:33.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7ffba9b52640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7ffba4074c30 con 0x7ffba4136270 2024-08-16T02:36:33.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7efcf1ced640 1 -- 172.21.15.133:0/3779039645 >> v1:172.21.15.133:6789/0 conn(0x7efcec09ba20 legacy=0x7efcec09be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7ffba37fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7ffba4133390 0x7ffba41337a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34292/0 (socket says 172.21.15.133:34292) 2024-08-16T02:36:33.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7ffba37fe640 1 -- 172.21.15.133:0/4277842365 learned_addr learned my addr 172.21.15.133:0/4277842365 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:33.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7efcf1ced640 1 -- 172.21.15.133:0/3779039645 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcec09d640 con 0x7efcec09c540 2024-08-16T02:36:33.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7ffba2ffd640 1 --2- 172.21.15.133:0/4277842365 >> v2:172.21.15.133:3300/0 conn(0x7ffba4136270 0x7ffba4138690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:33.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fecdeffd640 1 -- 172.21.15.133:0/3793712701 >> v1:172.21.15.133:6789/0 conn(0x7fece009a080 legacy=0x7fece009a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fecdeffd640 1 -- 172.21.15.133:0/3793712701 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece009ca40 con 0x7fece009aba0 2024-08-16T02:36:33.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fc877fff640 1 --2- 172.21.15.133:0/2188267116 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878138490 0x7fc87813a8b0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca27dfd4d733dee server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fecdeffd640 1 --2- 172.21.15.133:0/3793712701 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009aba0 0x7fece0136c80 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4fcde0a36bb5bf8 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7efcf1ced640 1 --2- 172.21.15.133:0/3779039645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09c540 0x7efcec13a720 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a18daeec35e69d server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7ffba2ffd640 1 -- 172.21.15.133:0/4277842365 >> v1:172.21.15.133:6789/0 conn(0x7ffba4133390 legacy=0x7ffba41337a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7ffba2ffd640 1 -- 172.21.15.133:0/4277842365 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba4074700 con 0x7ffba4136270 2024-08-16T02:36:33.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fc8777fe640 1 -- 172.21.15.133:0/2188267116 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc864009d90 con 0x7fc878138490 2024-08-16T02:36:33.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.980+0000 7fecde7fc640 1 -- 172.21.15.133:0/3793712701 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fecd4009d90 con 0x7fece009aba0 2024-08-16T02:36:33.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7fc8777fe640 1 -- 172.21.15.133:0/2188267116 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc86400e230 con 0x7fc878138490 2024-08-16T02:36:33.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7fc8777fe640 1 -- 172.21.15.133:0/2188267116 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc86400e4d0 con 0x7fc878138490 2024-08-16T02:36:33.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7efcf14ec640 1 -- 172.21.15.133:0/3779039645 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efcdc009d90 con 0x7efcec09c540 2024-08-16T02:36:33.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7fecde7fc640 1 -- 172.21.15.133:0/3793712701 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fecd400f470 con 0x7fece009aba0 2024-08-16T02:36:33.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7fecde7fc640 1 -- 172.21.15.133:0/3793712701 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fecd400f710 con 0x7fece009aba0 2024-08-16T02:36:33.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7efcf14ec640 1 -- 172.21.15.133:0/3779039645 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efcdc00e230 con 0x7efcec09c540 2024-08-16T02:36:33.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7efcf14ec640 1 -- 172.21.15.133:0/3779039645 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efcdc00e4d0 con 0x7efcec09c540 2024-08-16T02:36:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7ffba2ffd640 1 --2- 172.21.15.133:0/4277842365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4136270 0x7ffba4138690 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76515f42d74693fa server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7ffba27fc640 1 -- 172.21.15.133:0/4277842365 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ffb98009d90 con 0x7ffba4136270 2024-08-16T02:36:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7ffba27fc640 1 -- 172.21.15.133:0/4277842365 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ffb9800e230 con 0x7ffba4136270 2024-08-16T02:36:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.981+0000 7ffba27fc640 1 -- 172.21.15.133:0/4277842365 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ffb9800e4d0 con 0x7ffba4136270 2024-08-16T02:36:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fc87f0ed640 1 -- 172.21.15.133:0/2188267116 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878138490 msgr2=0x7fc87813a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fc87f0ed640 1 --2- 172.21.15.133:0/2188267116 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878138490 0x7fc87813a8b0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fc87f0ed640 1 -- 172.21.15.133:0/2188267116 shutdown_connections 2024-08-16T02:36:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fc87f0ed640 1 --2- 172.21.15.133:0/2188267116 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878138490 0x7fc87813a8b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fc87f0ed640 1 -- 172.21.15.133:0/2188267116 >> 172.21.15.133:0/2188267116 conn(0x7fc8780a5880 msgr2=0x7fc8780a5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7ffba9b52640 1 -- 172.21.15.133:0/4277842365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4136270 msgr2=0x7ffba4138690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7ffba9b52640 1 --2- 172.21.15.133:0/4277842365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4136270 0x7ffba4138690 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7ffba9b52640 1 -- 172.21.15.133:0/4277842365 shutdown_connections 2024-08-16T02:36:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fc87f0ed640 1 -- 172.21.15.133:0/2188267116 shutdown_connections 2024-08-16T02:36:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fc87f0ed640 1 -- 172.21.15.133:0/2188267116 wait complete. 2024-08-16T02:36:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7efcf4752640 1 -- 172.21.15.133:0/3779039645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09c540 msgr2=0x7efcec13a720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7efcf4752640 1 --2- 172.21.15.133:0/3779039645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09c540 0x7efcec13a720 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7ffba9b52640 1 --2- 172.21.15.133:0/4277842365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4136270 0x7ffba4138690 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7ffba9b52640 1 -- 172.21.15.133:0/4277842365 >> 172.21.15.133:0/4277842365 conn(0x7ffba40a2380 msgr2=0x7ffba40a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fece6112640 1 -- 172.21.15.133:0/3793712701 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009aba0 msgr2=0x7fece0136c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:33.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fece6112640 1 --2- 172.21.15.133:0/3793712701 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009aba0 0x7fece0136c80 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7ffba9b52640 1 -- 172.21.15.133:0/4277842365 shutdown_connections 2024-08-16T02:36:33.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fece6112640 1 -- 172.21.15.133:0/3793712701 shutdown_connections 2024-08-16T02:36:33.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fece6112640 1 --2- 172.21.15.133:0/3793712701 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009aba0 0x7fece0136c80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7ffba9b52640 1 -- 172.21.15.133:0/4277842365 wait complete. 2024-08-16T02:36:33.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fece6112640 1 -- 172.21.15.133:0/3793712701 >> 172.21.15.133:0/3793712701 conn(0x7fece00a0a10 msgr2=0x7fece00a0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7efcf4752640 1 -- 172.21.15.133:0/3779039645 shutdown_connections 2024-08-16T02:36:33.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7efcf4752640 1 --2- 172.21.15.133:0/3779039645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09c540 0x7efcec13a720 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:33.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fc87f0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:33.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fece6112640 1 -- 172.21.15.133:0/3793712701 shutdown_connections 2024-08-16T02:36:33.999 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7efcf4752640 1 -- 172.21.15.133:0/3779039645 >> 172.21.15.133:0/3779039645 conn(0x7efcec0a09c0 msgr2=0x7efcec0a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:33.999 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fece6112640 1 -- 172.21.15.133:0/3793712701 wait complete. 2024-08-16T02:36:33.999 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7efcf4752640 1 -- 172.21.15.133:0/3779039645 shutdown_connections 2024-08-16T02:36:33.999 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7efcf4752640 1 -- 172.21.15.133:0/3779039645 wait complete. 2024-08-16T02:36:34.000 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7ffba9b52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.000 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7fece6112640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.000 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.982+0000 7efcf4752640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.000 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.984+0000 7fc87f0ed640 1 Processor -- start 2024-08-16T02:36:34.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.984+0000 7ffba9b52640 1 Processor -- start 2024-08-16T02:36:34.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.984+0000 7efcf4752640 1 Processor -- start 2024-08-16T02:36:34.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.984+0000 7fc87f0ed640 1 -- start start 2024-08-16T02:36:34.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.984+0000 7fece6112640 1 Processor -- start 2024-08-16T02:36:34.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.984+0000 7efcf4752640 1 -- start start 2024-08-16T02:36:34.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.984+0000 7ffba9b52640 1 -- start start 2024-08-16T02:36:34.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.984+0000 7fece6112640 1 -- start start 2024-08-16T02:36:34.003 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7efcf4752640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09ba20 0x7efcec1a93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.003 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7efcf4752640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7efcec1acc80 con 0x7efcec09ba20 2024-08-16T02:36:34.003 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fc87f0ed640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878135490 0x7fc878131c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.004 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7ffba9b52640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4133390 0x7ffba411feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.004 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7ffba9b52640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7ffba4123be0 con 0x7ffba4133390 2024-08-16T02:36:34.004 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fece6112640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009a080 0x7fece01c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.004 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fece6112640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7fece0070980 con 0x7fece009a080 2024-08-16T02:36:34.004 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fc87f0ed640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7fc878121c40 con 0x7fc878135490 2024-08-16T02:36:34.005 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7efcf24ee640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09ba20 0x7efcec1a93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.005 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fc87ce89640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878135490 0x7fc878131c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.005 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fc87ce89640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878135490 0x7fc878131c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55982/0 (socket says 172.21.15.133:55982) 2024-08-16T02:36:34.005 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fc87ce89640 1 -- 172.21.15.133:0/1503977755 learned_addr learned my addr 172.21.15.133:0/1503977755 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.005 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7ffba37fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4133390 0x7ffba411feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.005 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7ffba37fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4133390 0x7ffba411feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55988/0 (socket says 172.21.15.133:55988) 2024-08-16T02:36:34.006 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fecdf7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009a080 0x7fece01c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.006 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7ffba37fe640 1 -- 172.21.15.133:0/3136497275 learned_addr learned my addr 172.21.15.133:0/3136497275 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.006 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fecdf7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009a080 0x7fece01c5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55990/0 (socket says 172.21.15.133:55990) 2024-08-16T02:36:34.007 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7efcf24ee640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09ba20 0x7efcec1a93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55972/0 (socket says 172.21.15.133:55972) 2024-08-16T02:36:34.007 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7efcf24ee640 1 -- 172.21.15.133:0/3276217592 learned_addr learned my addr 172.21.15.133:0/3276217592 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.007 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.985+0000 7fecdf7fe640 1 -- 172.21.15.133:0/3194765869 learned_addr learned my addr 172.21.15.133:0/3194765869 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.007 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7fc87ce89640 1 -- 172.21.15.133:0/1503977755 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87811fec0 con 0x7fc878135490 2024-08-16T02:36:34.007 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7ffba37fe640 1 -- 172.21.15.133:0/3136497275 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba4132ef0 con 0x7ffba4133390 2024-08-16T02:36:34.008 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7fc87ce89640 1 --2- 172.21.15.133:0/1503977755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878135490 0x7fc878131c80 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.008 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7efcf24ee640 1 -- 172.21.15.133:0/3276217592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcec1ab040 con 0x7efcec09ba20 2024-08-16T02:36:34.008 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7ffba37fe640 1 --2- 172.21.15.133:0/3136497275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4133390 0x7ffba411feb0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.009 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc86c005a60 con 0x7fc878135490 2024-08-16T02:36:34.009 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7fecdf7fe640 1 -- 172.21.15.133:0/3194765869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece0072df0 con 0x7fece009a080 2024-08-16T02:36:34.009 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc86c005bc0 con 0x7fc878135490 2024-08-16T02:36:34.009 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ffb94005a60 con 0x7ffba4133390 2024-08-16T02:36:34.010 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7efcf24ee640 1 --2- 172.21.15.133:0/3276217592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09ba20 0x7efcec1a93d0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.010 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc86c005e60 con 0x7fc878135490 2024-08-16T02:36:34.010 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ffb94005bc0 con 0x7ffba4133390 2024-08-16T02:36:34.010 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7ffba9b52640 1 -- 172.21.15.133:0/3136497275 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba4123e50 con 0x7ffba4133390 2024-08-16T02:36:34.011 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ffb94005e60 con 0x7ffba4133390 2024-08-16T02:36:34.011 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7ffba9b52640 1 -- 172.21.15.133:0/3136497275 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba41218f0 con 0x7ffba4133390 2024-08-16T02:36:34.011 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7fecdf7fe640 1 --2- 172.21.15.133:0/3194765869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009a080 0x7fece01c5cd0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.011 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efce8004b80 con 0x7efcec09ba20 2024-08-16T02:36:34.011 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.987+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc878134430 con 0x7fc878135490 2024-08-16T02:36:34.012 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efce8004ce0 con 0x7efcec09ba20 2024-08-16T02:36:34.012 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.986+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efce8007120 con 0x7efcec09ba20 2024-08-16T02:36:34.012 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.987+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcec1a8da0 con 0x7efcec09ba20 2024-08-16T02:36:34.012 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.987+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcec1ac130 con 0x7efcec09ba20 2024-08-16T02:36:34.012 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.987+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc878120ff0 con 0x7fc878135490 2024-08-16T02:36:34.013 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.987+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fecd0005a60 con 0x7fece009a080 2024-08-16T02:36:34.013 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.988+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efcec1a91d0 con 0x7efcec09ba20 2024-08-16T02:36:34.013 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.988+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece0073af0 con 0x7fece009a080 2024-08-16T02:36:34.013 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.988+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece0074af0 con 0x7fece009a080 2024-08-16T02:36:34.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.987+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffb7c000f80 con 0x7ffba4133390 2024-08-16T02:36:34.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.988+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc8780a01b0 con 0x7fc878135490 2024-08-16T02:36:34.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.988+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fecd0005bc0 con 0x7fece009a080 2024-08-16T02:36:34.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.988+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fece00745a0 con 0x7fece009a080 2024-08-16T02:36:34.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.988+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7efce8007280 con 0x7efcec09ba20 2024-08-16T02:36:34.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fecd0005e60 con 0x7fece009a080 2024-08-16T02:36:34.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fc86c00c7f0 con 0x7fc878135490 2024-08-16T02:36:34.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7efce2ffd640 1 --2- 172.21.15.133:0/3276217592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efcd003dd60 0x7efcd0040270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7efce804ea80 con 0x7efcec09ba20 2024-08-16T02:36:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7fc8757fa640 1 --2- 172.21.15.133:0/1503977755 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fc85c03dd60 0x7fc85c040270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fc86c04ea70 con 0x7fc878135490 2024-08-16T02:36:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7efce2ffd640 1 --2- 172.21.15.133:0/3276217592 >> v2:172.21.15.133:6804/2249864834 conn(0x7efcd0042d60 0x7efcd0045230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 --> v2:172.21.15.133:6804/2249864834 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efcd00458c0 con 0x7efcd0042d60 2024-08-16T02:36:34.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7fc8757fa640 1 --2- 172.21.15.133:0/1503977755 >> v2:172.21.15.133:6803/2214080839 conn(0x7fc85c042d60 0x7fc85c045230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 --> v2:172.21.15.133:6803/2214080839 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc85c0458c0 con 0x7fc85c042d60 2024-08-16T02:36:34.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7ffb9400c7f0 con 0x7ffba4133390 2024-08-16T02:36:34.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7efcf1ced640 1 --2- 172.21.15.133:0/3276217592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efcd003dd60 0x7efcd0040270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7efce800dc60 con 0x7efcec09ba20 2024-08-16T02:36:34.018 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7fc877fff640 1 --2- 172.21.15.133:0/1503977755 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fc85c03dd60 0x7fc85c040270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.018 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.989+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7fc86c0102a0 con 0x7fc878135490 2024-08-16T02:36:34.018 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7ffb8bfff640 1 --2- 172.21.15.133:0/3136497275 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ffb9003dd10 0x7ffb90040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.018 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7ffb9404e780 con 0x7ffba4133390 2024-08-16T02:36:34.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7fc87d68a640 1 --2- 172.21.15.133:0/1503977755 >> v2:172.21.15.133:6803/2214080839 conn(0x7fc85c042d60 0x7fc85c045230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:36:34.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7ffba2ffd640 1 --2- 172.21.15.133:0/3136497275 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ffb9003dd10 0x7ffb90040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7ffb8bfff640 1 --2- 172.21.15.133:0/3136497275 >> v2:172.21.15.137:6808/3166527287 conn(0x7ffb90042d10 0x7ffb900451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7efcf1ced640 1 --2- 172.21.15.133:0/3276217592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efcd003dd60 0x7efcd0040270 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7efcf2cef640 1 --2- 172.21.15.133:0/3276217592 >> v2:172.21.15.133:6804/2249864834 conn(0x7efcd0042d60 0x7efcd0045230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:36:34.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 --> v2:172.21.15.137:6808/3166527287 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb90045870 con 0x7ffb90042d10 2024-08-16T02:36:34.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fecd000c7f0 con 0x7fece009a080 2024-08-16T02:36:34.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.991+0000 7fc877fff640 1 --2- 172.21.15.133:0/1503977755 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fc85c03dd60 0x7fc85c040270 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.990+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7ffb9404eb60 con 0x7ffba4133390 2024-08-16T02:36:34.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.991+0000 7ffba2ffd640 1 --2- 172.21.15.133:0/3136497275 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ffb9003dd10 0x7ffb90040220 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.991+0000 7fecc7fff640 1 --2- 172.21.15.133:0/3194765869 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7feccc03dcc0 0x7feccc0401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.991+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fecd004ebb0 con 0x7fece009a080 2024-08-16T02:36:34.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.991+0000 7fecc7fff640 1 --2- 172.21.15.133:0/3194765869 >> v2:172.21.15.137:6805/34553420 conn(0x7feccc042cc0 0x7feccc045190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.991+0000 7fecdeffd640 1 --2- 172.21.15.133:0/3194765869 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7feccc03dcc0 0x7feccc0401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.991+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 --> v2:172.21.15.137:6805/34553420 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feccc045820 con 0x7feccc042cc0 2024-08-16T02:36:34.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.991+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7fecd004ef50 con 0x7fece009a080 2024-08-16T02:36:34.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.992+0000 7ffba3fff640 1 --2- 172.21.15.133:0/3136497275 >> v2:172.21.15.137:6808/3166527287 conn(0x7ffb90042d10 0x7ffb900451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:36:34.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.992+0000 7fecdeffd640 1 --2- 172.21.15.133:0/3194765869 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7feccc03dcc0 0x7feccc0401d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.992+0000 7fecdffff640 1 --2- 172.21.15.133:0/3194765869 >> v2:172.21.15.137:6805/34553420 conn(0x7feccc042cc0 0x7feccc045190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:36:34.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.994+0000 7efcf2cef640 1 --2- 172.21.15.133:0/3276217592 >> v2:172.21.15.133:6804/2249864834 conn(0x7efcd0042d60 0x7efcd0045230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.994+0000 7fc87d68a640 1 --2- 172.21.15.133:0/1503977755 >> v2:172.21.15.133:6803/2214080839 conn(0x7fc85c042d60 0x7fc85c045230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.996+0000 7fecdffff640 1 --2- 172.21.15.133:0/3194765869 >> v2:172.21.15.137:6805/34553420 conn(0x7feccc042cc0 0x7feccc045190 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.024 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.996+0000 7ffba3fff640 1 --2- 172.21.15.133:0/3136497275 >> v2:172.21.15.137:6808/3166527287 conn(0x7ffb90042d10 0x7ffb900451e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.024 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.996+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 <== osd.0 v2:172.21.15.133:6804/2249864834 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7efcd00458c0 con 0x7efcd0042d60 2024-08-16T02:36:34.024 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.996+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 <== osd.1 v2:172.21.15.133:6803/2214080839 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7fc85c0458c0 con 0x7fc85c042d60 2024-08-16T02:36:34.024 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.997+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 <== osd.3 v2:172.21.15.137:6805/34553420 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7feccc045820 con 0x7feccc042cc0 2024-08-16T02:36:34.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:33.997+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 <== osd.2 v2:172.21.15.137:6808/3166527287 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7ffb90045870 con 0x7ffb90042d10 2024-08-16T02:36:34.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.000+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 --> v2:172.21.15.133:6804/2249864834 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efcec09f920 con 0x7efcd0042d60 2024-08-16T02:36:34.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.001+0000 7efce2ffd640 1 -- 172.21.15.133:0/3276217592 <== osd.0 v2:172.21.15.133:6804/2249864834 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efce400a820 con 0x7efcd0042d60 2024-08-16T02:36:34.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 >> v2:172.21.15.133:6804/2249864834 conn(0x7efcd0042d60 msgr2=0x7efcd0045230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 --2- 172.21.15.133:0/3276217592 >> v2:172.21.15.133:6804/2249864834 conn(0x7efcd0042d60 0x7efcd0045230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efcd003dd60 msgr2=0x7efcd0040270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 --2- 172.21.15.133:0/3276217592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efcd003dd60 0x7efcd0040270 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09ba20 msgr2=0x7efcec1a93d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 --2- 172.21.15.133:0/3276217592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09ba20 0x7efcec1a93d0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 shutdown_connections 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 --2- 172.21.15.133:0/3276217592 >> v2:172.21.15.133:6804/2249864834 conn(0x7efcd0042d60 0x7efcd0045230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 --2- 172.21.15.133:0/3276217592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efcd003dd60 0x7efcd0040270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 --2- 172.21.15.133:0/3276217592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcec09ba20 0x7efcec1a93d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 >> 172.21.15.133:0/3276217592 conn(0x7efcec0a09c0 msgr2=0x7efcec0a27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 shutdown_connections 2024-08-16T02:36:34.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.002+0000 7efcf4752640 1 -- 172.21.15.133:0/3276217592 wait complete. 2024-08-16T02:36:34.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.003+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 --> v2:172.21.15.133:6803/2214080839 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc878073f70 con 0x7fc85c042d60 2024-08-16T02:36:34.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.003+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 --> v2:172.21.15.137:6808/3166527287 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ffb7c002c30 con 0x7ffb90042d10 2024-08-16T02:36:34.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.003+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 --> v2:172.21.15.137:6805/34553420 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fece009c730 con 0x7feccc042cc0 2024-08-16T02:36:34.029 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.004+0000 7ffb8bfff640 1 -- 172.21.15.133:0/3136497275 <== osd.2 v2:172.21.15.137:6808/3166527287 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ffb7c002c30 con 0x7ffb90042d10 2024-08-16T02:36:34.029 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.004+0000 7fc8757fa640 1 -- 172.21.15.133:0/1503977755 <== osd.1 v2:172.21.15.133:6803/2214080839 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc87000a820 con 0x7fc85c042d60 2024-08-16T02:36:34.029 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fecc7fff640 1 -- 172.21.15.133:0/3194765869 <== osd.3 v2:172.21.15.137:6805/34553420 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fecd800a820 con 0x7feccc042cc0 2024-08-16T02:36:34.030 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 >> v2:172.21.15.137:6808/3166527287 conn(0x7ffb90042d10 msgr2=0x7ffb900451e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.030 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 --2- 172.21.15.133:0/3136497275 >> v2:172.21.15.137:6808/3166527287 conn(0x7ffb90042d10 0x7ffb900451e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.030 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 >> v2:172.21.15.133:6803/2214080839 conn(0x7fc85c042d60 msgr2=0x7fc85c045230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.030 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 --2- 172.21.15.133:0/1503977755 >> v2:172.21.15.133:6803/2214080839 conn(0x7fc85c042d60 0x7fc85c045230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.031 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ffb9003dd10 msgr2=0x7ffb90040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.031 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 --2- 172.21.15.133:0/3136497275 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ffb9003dd10 0x7ffb90040220 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.031 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4133390 msgr2=0x7ffba411feb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 --2- 172.21.15.133:0/3136497275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4133390 0x7ffba411feb0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 shutdown_connections 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 --2- 172.21.15.133:0/3136497275 >> v2:172.21.15.137:6808/3166527287 conn(0x7ffb90042d10 0x7ffb900451e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 --2- 172.21.15.133:0/3136497275 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ffb9003dd10 0x7ffb90040220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 --2- 172.21.15.133:0/3136497275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffba4133390 0x7ffba411feb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fc85c03dd60 msgr2=0x7fc85c040270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 --2- 172.21.15.133:0/1503977755 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fc85c03dd60 0x7fc85c040270 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878135490 msgr2=0x7fc878131c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 --2- 172.21.15.133:0/1503977755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878135490 0x7fc878131c80 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.033 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 >> 172.21.15.133:0/3136497275 conn(0x7ffba40a2380 msgr2=0x7ffba4134050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.033 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 >> v2:172.21.15.137:6805/34553420 conn(0x7feccc042cc0 msgr2=0x7feccc045190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.033 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fece6112640 1 --2- 172.21.15.133:0/3194765869 >> v2:172.21.15.137:6805/34553420 conn(0x7feccc042cc0 0x7feccc045190 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.033 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7feccc03dcc0 msgr2=0x7feccc0401d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.033 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fece6112640 1 --2- 172.21.15.133:0/3194765869 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7feccc03dcc0 0x7feccc0401d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.033 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009a080 msgr2=0x7fece01c5cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 shutdown_connections 2024-08-16T02:36:34.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fece6112640 1 --2- 172.21.15.133:0/3194765869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009a080 0x7fece01c5cd0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 shutdown_connections 2024-08-16T02:36:34.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 --2- 172.21.15.133:0/1503977755 >> v2:172.21.15.133:6803/2214080839 conn(0x7fc85c042d60 0x7fc85c045230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 --2- 172.21.15.133:0/1503977755 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fc85c03dd60 0x7fc85c040270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.005+0000 7fc87f0ed640 1 --2- 172.21.15.133:0/1503977755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc878135490 0x7fc878131c80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 >> 172.21.15.133:0/1503977755 conn(0x7fc8780a5880 msgr2=0x7fc8781bd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7ffb89ffb640 1 -- 172.21.15.133:0/3136497275 wait complete. 2024-08-16T02:36:34.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 shutdown_connections 2024-08-16T02:36:34.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fc87f0ed640 1 -- 172.21.15.133:0/1503977755 wait complete. 2024-08-16T02:36:34.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 shutdown_connections 2024-08-16T02:36:34.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fece6112640 1 --2- 172.21.15.133:0/3194765869 >> v2:172.21.15.137:6805/34553420 conn(0x7feccc042cc0 0x7feccc045190 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fece6112640 1 --2- 172.21.15.133:0/3194765869 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7feccc03dcc0 0x7feccc0401d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fece6112640 1 --2- 172.21.15.133:0/3194765869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fece009a080 0x7fece01c5cd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 >> 172.21.15.133:0/3194765869 conn(0x7fece00a0a10 msgr2=0x7fece009e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 shutdown_connections 2024-08-16T02:36:34.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.006+0000 7fece6112640 1 -- 172.21.15.133:0/3194765869 wait complete. 2024-08-16T02:36:34.037 INFO:teuthology.orchestra.run.smithi133.stdout:38654705672 2024-08-16T02:36:34.037 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-16T02:36:34.038 INFO:teuthology.orchestra.run.smithi133.stdout:38654705672 2024-08-16T02:36:34.038 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-16T02:36:34.039 INFO:teuthology.orchestra.run.smithi133.stdout:38654705672 2024-08-16T02:36:34.039 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-16T02:36:34.040 INFO:teuthology.orchestra.run.smithi133.stdout:38654705672 2024-08-16T02:36:34.040 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-16T02:36:34.199 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.193+0000 7efc81c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.199 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.193+0000 7ff60966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.200 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.199+0000 7efc81c64640 1 Processor -- start 2024-08-16T02:36:34.200 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.199+0000 7efc81c64640 1 -- start start 2024-08-16T02:36:34.201 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.200+0000 7ff60966d640 1 Processor -- start 2024-08-16T02:36:34.201 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.200+0000 7ff60966d640 1 -- start start 2024-08-16T02:36:34.201 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.200+0000 7efc81c64640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7efc7c071bd0 0x7efc7c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.202 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.200+0000 7efc81c64640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc7c074580 con 0x7efc7c072520 2024-08-16T02:36:34.202 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7efc81c64640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7efc7c073df0 con 0x7efc7c071bd0 2024-08-16T02:36:34.202 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7efc7b7fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7efc7c071bd0 0x7efc7c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.202 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7efc7affd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7efc7c072520 0x7efc7c131f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34308/0 (socket says 172.21.15.133:34308) 2024-08-16T02:36:34.202 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7efc7affd640 1 -- 172.21.15.133:0/633164193 learned_addr learned my addr 172.21.15.133:0/633164193 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:34.202 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7ff60966d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7ff6040726f0 0x7ff604138540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.202 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7ff60966d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff604074580 con 0x7ff604071bd0 2024-08-16T02:36:34.203 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7ff60966d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7ff604073df0 con 0x7ff6040726f0 2024-08-16T02:36:34.203 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7efc7b7fe640 1 -- 172.21.15.133:0/633164193 >> v1:172.21.15.133:6789/0 conn(0x7efc7c072520 legacy=0x7efc7c131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.203 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7efc7b7fe640 1 -- 172.21.15.133:0/633164193 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7c074070 con 0x7efc7c071bd0 2024-08-16T02:36:34.203 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7ff602ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7ff604071bd0 0x7ff604071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34318/0 (socket says 172.21.15.133:34318) 2024-08-16T02:36:34.204 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7ff602ffd640 1 -- 172.21.15.133:0/2933200175 learned_addr learned my addr 172.21.15.133:0/2933200175 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:34.204 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7ff6027fc640 1 --2- 172.21.15.133:0/2933200175 >> v2:172.21.15.133:3300/0 conn(0x7ff6040726f0 0x7ff604138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.204 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7efc7b7fe640 1 --2- 172.21.15.133:0/633164193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c071fe0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af22bff76863cb6 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.204 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.201+0000 7efc7a7fc640 1 -- 172.21.15.133:0/633164193 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efc6c009d90 con 0x7efc7c071bd0 2024-08-16T02:36:34.204 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.202+0000 7ff6027fc640 1 -- 172.21.15.133:0/2933200175 >> v1:172.21.15.133:6789/0 conn(0x7ff604071bd0 legacy=0x7ff604071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.205 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.202+0000 7efc7a7fc640 1 -- 172.21.15.133:0/633164193 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efc6c00e230 con 0x7efc7c071bd0 2024-08-16T02:36:34.205 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.202+0000 7efc7a7fc640 1 -- 172.21.15.133:0/633164193 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efc6c00e4d0 con 0x7efc7c071bd0 2024-08-16T02:36:34.206 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.202+0000 7ff6027fc640 1 -- 172.21.15.133:0/2933200175 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff604074070 con 0x7ff6040726f0 2024-08-16T02:36:34.206 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.202+0000 7ff6027fc640 1 --2- 172.21.15.133:0/2933200175 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6040726f0 0x7ff604138540 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233a6bd91413df29 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.206 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.202+0000 7ff601ffb640 1 -- 172.21.15.133:0/2933200175 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff5f8009d90 con 0x7ff6040726f0 2024-08-16T02:36:34.206 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.202+0000 7ff601ffb640 1 -- 172.21.15.133:0/2933200175 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff5f800e230 con 0x7ff6040726f0 2024-08-16T02:36:34.206 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.202+0000 7ff601ffb640 1 -- 172.21.15.133:0/2933200175 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff5f800e4d0 con 0x7ff6040726f0 2024-08-16T02:36:34.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7efc81c64640 1 -- 172.21.15.133:0/633164193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 msgr2=0x7efc7c071fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7efc81c64640 1 --2- 172.21.15.133:0/633164193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c071fe0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7efc81c64640 1 -- 172.21.15.133:0/633164193 shutdown_connections 2024-08-16T02:36:34.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7efc81c64640 1 --2- 172.21.15.133:0/633164193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c071fe0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7efc81c64640 1 -- 172.21.15.133:0/633164193 >> 172.21.15.133:0/633164193 conn(0x7efc7c0a2480 msgr2=0x7efc7c0a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7efc81c64640 1 -- 172.21.15.133:0/633164193 shutdown_connections 2024-08-16T02:36:34.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7efc81c64640 1 -- 172.21.15.133:0/633164193 wait complete. 2024-08-16T02:36:34.208 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7ff60966d640 1 -- 172.21.15.133:0/2933200175 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6040726f0 msgr2=0x7ff604138540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.208 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7ff60966d640 1 --2- 172.21.15.133:0/2933200175 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6040726f0 0x7ff604138540 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.208 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7ff60966d640 1 -- 172.21.15.133:0/2933200175 shutdown_connections 2024-08-16T02:36:34.209 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7ff60966d640 1 --2- 172.21.15.133:0/2933200175 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6040726f0 0x7ff604138540 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.209 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7ff60966d640 1 -- 172.21.15.133:0/2933200175 >> 172.21.15.133:0/2933200175 conn(0x7ff6040a2480 msgr2=0x7ff6040a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.209 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7ff60966d640 1 -- 172.21.15.133:0/2933200175 shutdown_connections 2024-08-16T02:36:34.209 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7ff60966d640 1 -- 172.21.15.133:0/2933200175 wait complete. 2024-08-16T02:36:34.209 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7efc81c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.209 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.203+0000 7ff60966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.210 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.193+0000 7f0091e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.210 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.193+0000 7f7e54392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.210 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.204+0000 7f0091e2d640 1 Processor -- start 2024-08-16T02:36:34.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.204+0000 7f7e54392640 1 Processor -- start 2024-08-16T02:36:34.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.204+0000 7efc81c64640 1 Processor -- start 2024-08-16T02:36:34.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.204+0000 7f0091e2d640 1 -- start start 2024-08-16T02:36:34.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.204+0000 7f7e54392640 1 -- start start 2024-08-16T02:36:34.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.205+0000 7efc81c64640 1 -- start start 2024-08-16T02:36:34.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.205+0000 7ff60966d640 1 Processor -- start 2024-08-16T02:36:34.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.205+0000 7ff60966d640 1 -- start start 2024-08-16T02:36:34.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f0091e2d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f008c0726f0 0x7f008c138540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f0091e2d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f008c074580 con 0x7f008c071bd0 2024-08-16T02:36:34.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f0091e2d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f008c073df0 con 0x7f008c0726f0 2024-08-16T02:36:34.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7efc81c64640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c09d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7efc81c64640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c122010 con 0x7efc7c071bd0 2024-08-16T02:36:34.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f008b7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f008c071bd0 0x7f008c071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34334/0 (socket says 172.21.15.133:34334) 2024-08-16T02:36:34.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f7e54392640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f7e4c0726f0 0x7f7e4c138540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f7e54392640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e4c074580 con 0x7f7e4c071bd0 2024-08-16T02:36:34.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f7e54392640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f7e4c073df0 con 0x7f7e4c0726f0 2024-08-16T02:36:34.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f008b7fe640 1 -- 172.21.15.133:0/2529073500 learned_addr learned my addr 172.21.15.133:0/2529073500 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:34.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f7e5212e640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f7e4c071bd0 0x7f7e4c071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34350/0 (socket says 172.21.15.133:34350) 2024-08-16T02:36:34.215 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f7e5212e640 1 -- 172.21.15.133:0/1977401020 learned_addr learned my addr 172.21.15.133:0/1977401020 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:34.215 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f7e5192d640 1 --2- 172.21.15.133:0/1977401020 >> v2:172.21.15.133:3300/0 conn(0x7f7e4c0726f0 0x7f7e4c138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.215 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f008affd640 1 --2- 172.21.15.133:0/2529073500 >> v2:172.21.15.133:3300/0 conn(0x7f008c0726f0 0x7f008c138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.215 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7efc7b7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c09d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.215 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7efc7b7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c09d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56028/0 (socket says 172.21.15.133:56028) 2024-08-16T02:36:34.215 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7efc7b7fe640 1 -- 172.21.15.133:0/1690729427 learned_addr learned my addr 172.21.15.133:0/1690729427 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.216 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7ff60966d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff604071bd0 0x7ff60406f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.216 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f008a7fc640 1 -- 172.21.15.133:0/2529073500 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84822817 0 0) 0x7f008c074580 con 0x7f008c071bd0 2024-08-16T02:36:34.216 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f008a7fc640 1 -- 172.21.15.133:0/2529073500 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0074003580 con 0x7f008c071bd0 2024-08-16T02:36:34.216 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7ff60966d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7ff604124e80 con 0x7ff604071bd0 2024-08-16T02:36:34.217 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f7e5192d640 1 -- 172.21.15.133:0/1977401020 >> v1:172.21.15.133:6789/0 conn(0x7f7e4c071bd0 legacy=0x7f7e4c071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.217 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.206+0000 7f7e5192d640 1 -- 172.21.15.133:0/1977401020 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4c074070 con 0x7f7e4c0726f0 2024-08-16T02:36:34.217 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7efc7b7fe640 1 -- 172.21.15.133:0/1690729427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7c09ed30 con 0x7efc7c071bd0 2024-08-16T02:36:34.218 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f008affd640 1 -- 172.21.15.133:0/2529073500 >> v1:172.21.15.133:6789/0 conn(0x7f008c071bd0 legacy=0x7f008c071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.218 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7efc7b7fe640 1 --2- 172.21.15.133:0/1690729427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c09d9c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.218 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f008affd640 1 -- 172.21.15.133:0/2529073500 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008c074070 con 0x7f008c0726f0 2024-08-16T02:36:34.218 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7ff602ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff604071bd0 0x7ff60406f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.219 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7ff602ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff604071bd0 0x7ff60406f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56044/0 (socket says 172.21.15.133:56044) 2024-08-16T02:36:34.219 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7ff602ffd640 1 -- 172.21.15.133:0/57094894 learned_addr learned my addr 172.21.15.133:0/57094894 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.219 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f7e5192d640 1 --2- 172.21.15.133:0/1977401020 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c0726f0 0x7f7e4c138540 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b299282f7fd8b09e server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.219 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f008affd640 1 --2- 172.21.15.133:0/2529073500 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c0726f0 0x7f008c138540 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f474b2d2e14176e1 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.220 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7efc63fff640 1 -- 172.21.15.133:0/1690729427 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efc6c014620 con 0x7efc7c071bd0 2024-08-16T02:36:34.220 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7efc63fff640 1 -- 172.21.15.133:0/1690729427 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efc6c014780 con 0x7efc7c071bd0 2024-08-16T02:36:34.220 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7efc63fff640 1 -- 172.21.15.133:0/1690729427 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efc6c014a20 con 0x7efc7c071bd0 2024-08-16T02:36:34.220 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc7c132220 con 0x7efc7c071bd0 2024-08-16T02:36:34.220 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f7e5112c640 1 -- 172.21.15.133:0/1977401020 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7e3c009d90 con 0x7f7e4c0726f0 2024-08-16T02:36:34.220 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7ff602ffd640 1 -- 172.21.15.133:0/57094894 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff604124810 con 0x7ff604071bd0 2024-08-16T02:36:34.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f7e5112c640 1 -- 172.21.15.133:0/1977401020 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f7e3c00e230 con 0x7f7e4c0726f0 2024-08-16T02:36:34.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f7e5112c640 1 -- 172.21.15.133:0/1977401020 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7e3c009d90 con 0x7f7e4c0726f0 2024-08-16T02:36:34.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f008a7fc640 1 -- 172.21.15.133:0/2529073500 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0080009d90 con 0x7f008c0726f0 2024-08-16T02:36:34.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc7c1207f0 con 0x7efc7c071bd0 2024-08-16T02:36:34.222 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f008a7fc640 1 -- 172.21.15.133:0/2529073500 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f008000e230 con 0x7f008c0726f0 2024-08-16T02:36:34.222 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7f008a7fc640 1 -- 172.21.15.133:0/2529073500 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f008000e4d0 con 0x7f008c0726f0 2024-08-16T02:36:34.223 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.207+0000 7ff602ffd640 1 --2- 172.21.15.133:0/57094894 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff604071bd0 0x7ff60406f9c0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.223 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.208+0000 7ff5eb7fe640 1 -- 172.21.15.133:0/57094894 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff5f4005a60 con 0x7ff604071bd0 2024-08-16T02:36:34.223 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.208+0000 7ff5eb7fe640 1 -- 172.21.15.133:0/57094894 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff5f4005bc0 con 0x7ff604071bd0 2024-08-16T02:36:34.223 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.208+0000 7ff5eb7fe640 1 -- 172.21.15.133:0/57094894 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff5f4005e60 con 0x7ff604071bd0 2024-08-16T02:36:34.223 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.208+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff604125120 con 0x7ff604071bd0 2024-08-16T02:36:34.223 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc7c06a020 con 0x7efc7c071bd0 2024-08-16T02:36:34.223 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7efc63fff640 1 -- 172.21.15.133:0/1690729427 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7efc6c014bd0 con 0x7efc7c071bd0 2024-08-16T02:36:34.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f0091e2d640 1 -- 172.21.15.133:0/2529073500 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c0726f0 msgr2=0x7f008c138540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f0091e2d640 1 --2- 172.21.15.133:0/2529073500 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c0726f0 0x7f008c138540 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.208+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6041240b0 con 0x7ff604071bd0 2024-08-16T02:36:34.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f7e54392640 1 -- 172.21.15.133:0/1977401020 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c0726f0 msgr2=0x7f7e4c138540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.225 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7efc63fff640 1 --2- 172.21.15.133:0/1690729427 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efc6803dd10 0x7efc68040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.225 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f7e54392640 1 --2- 172.21.15.133:0/1977401020 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c0726f0 0x7f7e4c138540 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.225 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f7e54392640 1 -- 172.21.15.133:0/1977401020 shutdown_connections 2024-08-16T02:36:34.225 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f7e54392640 1 --2- 172.21.15.133:0/1977401020 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c0726f0 0x7f7e4c138540 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.225 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f7e54392640 1 -- 172.21.15.133:0/1977401020 >> 172.21.15.133:0/1977401020 conn(0x7f7e4c0a2480 msgr2=0x7f7e4c0a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.225 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7efc63fff640 1 -- 172.21.15.133:0/1690729427 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7efc6c019040 con 0x7efc7c071bd0 2024-08-16T02:36:34.226 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f7e54392640 1 -- 172.21.15.133:0/1977401020 shutdown_connections 2024-08-16T02:36:34.226 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f0091e2d640 1 -- 172.21.15.133:0/2529073500 shutdown_connections 2024-08-16T02:36:34.226 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f0091e2d640 1 --2- 172.21.15.133:0/2529073500 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c0726f0 0x7f008c138540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.226 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f0091e2d640 1 -- 172.21.15.133:0/2529073500 >> 172.21.15.133:0/2529073500 conn(0x7f008c0a2480 msgr2=0x7f008c0a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.227 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f7e54392640 1 -- 172.21.15.133:0/1977401020 wait complete. 2024-08-16T02:36:34.227 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f0091e2d640 1 -- 172.21.15.133:0/2529073500 shutdown_connections 2024-08-16T02:36:34.227 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7f0091e2d640 1 -- 172.21.15.133:0/2529073500 wait complete. 2024-08-16T02:36:34.227 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.209+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff604124560 con 0x7ff604071bd0 2024-08-16T02:36:34.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.210+0000 7f7e54392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.210+0000 7efc7affd640 1 --2- 172.21.15.133:0/1690729427 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efc6803dd10 0x7efc68040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.210+0000 7f0091e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.229 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.210+0000 7ff5eb7fe640 1 -- 172.21.15.133:0/57094894 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7ff5f400c7f0 con 0x7ff604071bd0 2024-08-16T02:36:34.229 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.210+0000 7ff5eb7fe640 1 --2- 172.21.15.133:0/57094894 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ff5f003dd60 0x7ff5f0040270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.229 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.210+0000 7ff5eb7fe640 1 -- 172.21.15.133:0/57094894 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7ff5f404e820 con 0x7ff604071bd0 2024-08-16T02:36:34.229 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.210+0000 7ff6027fc640 1 --2- 172.21.15.133:0/57094894 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ff5f003dd60 0x7ff5f0040270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.229 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.210+0000 7efc7affd640 1 --2- 172.21.15.133:0/1690729427 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efc6803dd10 0x7efc68040220 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.211+0000 7ff6027fc640 1 --2- 172.21.15.133:0/57094894 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ff5f003dd60 0x7ff5f0040270 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.211+0000 7f0091e2d640 1 Processor -- start 2024-08-16T02:36:34.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.211+0000 7f7e54392640 1 Processor -- start 2024-08-16T02:36:34.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.211+0000 7f7e54392640 1 -- start start 2024-08-16T02:36:34.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.211+0000 7f0091e2d640 1 -- start start 2024-08-16T02:36:34.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f7e54392640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c071bd0 0x7f7e4c1203a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f7e54392640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c132ed0 con 0x7f7e4c071bd0 2024-08-16T02:36:34.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f0091e2d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c071bd0 0x7f008c1201a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f0091e2d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f008c123810 con 0x7f008c071bd0 2024-08-16T02:36:34.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f7e5212e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c071bd0 0x7f7e4c1203a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f7e5212e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c071bd0 0x7f7e4c1203a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56046/0 (socket says 172.21.15.133:56046) 2024-08-16T02:36:34.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f7e5212e640 1 -- 172.21.15.133:0/1200596651 learned_addr learned my addr 172.21.15.133:0/1200596651 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f008b7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c071bd0 0x7f008c1201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.212+0000 7f008b7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c071bd0 0x7f008c1201a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56058/0 (socket says 172.21.15.133:56058) 2024-08-16T02:36:34.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.213+0000 7f008b7fe640 1 -- 172.21.15.133:0/3244336320 learned_addr learned my addr 172.21.15.133:0/3244336320 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.213+0000 7f7e5212e640 1 -- 172.21.15.133:0/1200596651 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4c11f9c0 con 0x7f7e4c071bd0 2024-08-16T02:36:34.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.213+0000 7f008b7fe640 1 -- 172.21.15.133:0/3244336320 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008c11fea0 con 0x7f008c071bd0 2024-08-16T02:36:34.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.213+0000 7f7e5212e640 1 --2- 172.21.15.133:0/1200596651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c071bd0 0x7f7e4c1203a0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.213+0000 7f008b7fe640 1 --2- 172.21.15.133:0/3244336320 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c071bd0 0x7f008c1201a0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.215+0000 7efc63fff640 1 -- 172.21.15.133:0/1690729427 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7efc6c018970 con 0x7efc7c071bd0 2024-08-16T02:36:34.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f0073fff640 1 -- 172.21.15.133:0/3244336320 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f007c005a60 con 0x7f008c071bd0 2024-08-16T02:36:34.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f7e42ffd640 1 -- 172.21.15.133:0/1200596651 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7e48005a60 con 0x7f7e4c071bd0 2024-08-16T02:36:34.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f7e42ffd640 1 -- 172.21.15.133:0/1200596651 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f7e48005bc0 con 0x7f7e4c071bd0 2024-08-16T02:36:34.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f0073fff640 1 -- 172.21.15.133:0/3244336320 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f007c005bc0 con 0x7f008c071bd0 2024-08-16T02:36:34.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f0073fff640 1 -- 172.21.15.133:0/3244336320 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f007c005e60 con 0x7f008c071bd0 2024-08-16T02:36:34.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f7e42ffd640 1 -- 172.21.15.133:0/1200596651 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7e48005e60 con 0x7f7e4c071bd0 2024-08-16T02:36:34.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4c11ff70 con 0x7f7e4c071bd0 2024-08-16T02:36:34.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008c121290 con 0x7f008c071bd0 2024-08-16T02:36:34.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7ff5eb7fe640 1 -- 172.21.15.133:0/57094894 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7ff5f400d7f0 con 0x7ff604071bd0 2024-08-16T02:36:34.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f008c11fb90 con 0x7f008c071bd0 2024-08-16T02:36:34.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.220+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e4c122090 con 0x7f7e4c071bd0 2024-08-16T02:36:34.237 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.222+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f008c06a020 con 0x7f008c071bd0 2024-08-16T02:36:34.237 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.222+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e4c06a020 con 0x7f7e4c071bd0 2024-08-16T02:36:34.237 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.222+0000 7f7e42ffd640 1 -- 172.21.15.133:0/1200596651 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f7e480070b0 con 0x7f7e4c071bd0 2024-08-16T02:36:34.237 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.222+0000 7f7e42ffd640 1 --2- 172.21.15.133:0/1200596651 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f7e3003dd60 0x7f7e30040270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.237 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.222+0000 7f7e42ffd640 1 -- 172.21.15.133:0/1200596651 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f7e4804e760 con 0x7f7e4c071bd0 2024-08-16T02:36:34.238 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.222+0000 7f0073fff640 1 -- 172.21.15.133:0/3244336320 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f007c0070b0 con 0x7f008c071bd0 2024-08-16T02:36:34.238 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.222+0000 7f7e5192d640 1 --2- 172.21.15.133:0/1200596651 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f7e3003dd60 0x7f7e30040270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.238 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.223+0000 7f0073fff640 1 --2- 172.21.15.133:0/3244336320 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f007803dd10 0x7f0078040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.239 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.223+0000 7f0073fff640 1 -- 172.21.15.133:0/3244336320 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f007c04e720 con 0x7f008c071bd0 2024-08-16T02:36:34.239 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.223+0000 7f7e5192d640 1 --2- 172.21.15.133:0/1200596651 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f7e3003dd60 0x7f7e30040270 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.239 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.223+0000 7f008affd640 1 --2- 172.21.15.133:0/3244336320 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f007803dd10 0x7f0078040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.239 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.223+0000 7f008affd640 1 --2- 172.21.15.133:0/3244336320 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f007803dd10 0x7f0078040220 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.240 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.227+0000 7f7e42ffd640 1 -- 172.21.15.133:0/1200596651 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f7e4800d7f0 con 0x7f7e4c071bd0 2024-08-16T02:36:34.240 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.238+0000 7f0073fff640 1 -- 172.21.15.133:0/3244336320 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f007c00d840 con 0x7f008c071bd0 2024-08-16T02:36:34.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.415+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff60406a020 con 0x7ff604071bd0 2024-08-16T02:36:34.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.415+0000 7ff5eb7fe640 1 -- 172.21.15.133:0/57094894 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7ff5f400d610 con 0x7ff604071bd0 2024-08-16T02:36:34.416 INFO:teuthology.orchestra.run.smithi133.stdout:38654705672 2024-08-16T02:36:34.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ff5f003dd60 msgr2=0x7ff5f0040270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 --2- 172.21.15.133:0/57094894 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ff5f003dd60 0x7ff5f0040270 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff604071bd0 msgr2=0x7ff60406f9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 --2- 172.21.15.133:0/57094894 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff604071bd0 0x7ff60406f9c0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 shutdown_connections 2024-08-16T02:36:34.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 --2- 172.21.15.133:0/57094894 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7ff5f003dd60 0x7ff5f0040270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 --2- 172.21.15.133:0/57094894 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff604071bd0 0x7ff60406f9c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 >> 172.21.15.133:0/57094894 conn(0x7ff6040a2480 msgr2=0x7ff60409c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 shutdown_connections 2024-08-16T02:36:34.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.417+0000 7ff60966d640 1 -- 172.21.15.133:0/57094894 wait complete. 2024-08-16T02:36:34.426 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.426+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7efc7c0a17a0 con 0x7efc7c071bd0 2024-08-16T02:36:34.427 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.426+0000 7efc63fff640 1 -- 172.21.15.133:0/1690729427 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7efc6c018190 con 0x7efc7c071bd0 2024-08-16T02:36:34.427 INFO:teuthology.orchestra.run.smithi133.stdout:38654705672 2024-08-16T02:36:34.427 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.427+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7e4c121c70 con 0x7f7e4c071bd0 2024-08-16T02:36:34.428 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.427+0000 7f7e42ffd640 1 -- 172.21.15.133:0/1200596651 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7f7e4800d610 con 0x7f7e4c071bd0 2024-08-16T02:36:34.428 INFO:teuthology.orchestra.run.smithi133.stdout:38654705672 2024-08-16T02:36:34.429 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.1 2024-08-16T02:36:34.430 DEBUG:teuthology.parallel:result is None 2024-08-16T02:36:34.430 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efc6803dd10 msgr2=0x7efc68040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.431 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 --2- 172.21.15.133:0/1690729427 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efc6803dd10 0x7efc68040220 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.431 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 msgr2=0x7efc7c09d9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.431 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 --2- 172.21.15.133:0/1690729427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c09d9c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.432 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f7e3003dd60 msgr2=0x7f7e30040270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.432 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 --2- 172.21.15.133:0/1200596651 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f7e3003dd60 0x7f7e30040270 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.432 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c071bd0 msgr2=0x7f7e4c1203a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.432 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 --2- 172.21.15.133:0/1200596651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c071bd0 0x7f7e4c1203a0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.432 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 shutdown_connections 2024-08-16T02:36:34.432 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 --2- 172.21.15.133:0/1690729427 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7efc6803dd10 0x7efc68040220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.433 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 shutdown_connections 2024-08-16T02:36:34.433 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 --2- 172.21.15.133:0/1200596651 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f7e3003dd60 0x7f7e30040270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.433 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 --2- 172.21.15.133:0/1200596651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e4c071bd0 0x7f7e4c1203a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.433 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 >> 172.21.15.133:0/1200596651 conn(0x7f7e4c0a2480 msgr2=0x7f7e4c09c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.433 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 --2- 172.21.15.133:0/1690729427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc7c071bd0 0x7efc7c09d9c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.433 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 >> 172.21.15.133:0/1690729427 conn(0x7efc7c0a2480 msgr2=0x7efc7c070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 shutdown_connections 2024-08-16T02:36:34.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 shutdown_connections 2024-08-16T02:36:34.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7f7e54392640 1 -- 172.21.15.133:0/1200596651 wait complete. 2024-08-16T02:36:34.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.430+0000 7efc81c64640 1 -- 172.21.15.133:0/1690729427 wait complete. 2024-08-16T02:36:34.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.3 2024-08-16T02:36:34.442 DEBUG:teuthology.parallel:result is None 2024-08-16T02:36:34.447 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.0 2024-08-16T02:36:34.447 DEBUG:teuthology.parallel:result is None 2024-08-16T02:36:34.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.452+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f008c0647b0 con 0x7f008c071bd0 2024-08-16T02:36:34.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.452+0000 7f0073fff640 1 -- 172.21.15.133:0/3244336320 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7f007c00d660 con 0x7f008c071bd0 2024-08-16T02:36:34.453 INFO:teuthology.orchestra.run.smithi133.stdout:38654705672 2024-08-16T02:36:34.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.454+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f007803dd10 msgr2=0x7f0078040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.454+0000 7f0091e2d640 1 --2- 172.21.15.133:0/3244336320 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f007803dd10 0x7f0078040220 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.454+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c071bd0 msgr2=0x7f008c1201a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.454+0000 7f0091e2d640 1 --2- 172.21.15.133:0/3244336320 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c071bd0 0x7f008c1201a0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.455+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 shutdown_connections 2024-08-16T02:36:34.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.455+0000 7f0091e2d640 1 --2- 172.21.15.133:0/3244336320 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f007803dd10 0x7f0078040220 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.455+0000 7f0091e2d640 1 --2- 172.21.15.133:0/3244336320 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f008c071bd0 0x7f008c1201a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.455+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 >> 172.21.15.133:0/3244336320 conn(0x7f008c0a2480 msgr2=0x7f008c09c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.455+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 shutdown_connections 2024-08-16T02:36:34.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.455+0000 7f0091e2d640 1 -- 172.21.15.133:0/3244336320 wait complete. 2024-08-16T02:36:34.467 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.2 2024-08-16T02:36:34.467 DEBUG:teuthology.parallel:result is None 2024-08-16T02:36:34.467 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-16T02:36:34.467 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:36:34.614 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.609+0000 7f29f202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.615 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.614+0000 7f29f202d640 1 Processor -- start 2024-08-16T02:36:34.615 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.614+0000 7f29f202d640 1 -- start start 2024-08-16T02:36:34.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.615+0000 7f29f202d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f29ec135150 0x7f29ec137570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.615+0000 7f29f202d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ec09ca30 con 0x7f29ec1322e0 2024-08-16T02:36:34.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.615+0000 7f29f202d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f29ec09e6b0 con 0x7f29ec135150 2024-08-16T02:36:34.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.615+0000 7f29eb7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f29ec1322e0 0x7f29ec1326f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34352/0 (socket says 172.21.15.133:34352) 2024-08-16T02:36:34.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.615+0000 7f29eb7fe640 1 -- 172.21.15.133:0/7359724 learned_addr learned my addr 172.21.15.133:0/7359724 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:34.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.615+0000 7f29eaffd640 1 --2- 172.21.15.133:0/7359724 >> v2:172.21.15.133:3300/0 conn(0x7f29ec135150 0x7f29ec137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.616+0000 7f29ea7fc640 1 -- 172.21.15.133:0/7359724 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252146958 0 0) 0x7f29ec09ca30 con 0x7f29ec1322e0 2024-08-16T02:36:34.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.616+0000 7f29ea7fc640 1 -- 172.21.15.133:0/7359724 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29d4003580 con 0x7f29ec1322e0 2024-08-16T02:36:34.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.616+0000 7f29eaffd640 1 -- 172.21.15.133:0/7359724 >> v1:172.21.15.133:6789/0 conn(0x7f29ec1322e0 legacy=0x7f29ec1326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.616+0000 7f29eaffd640 1 -- 172.21.15.133:0/7359724 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec09e3a0 con 0x7f29ec135150 2024-08-16T02:36:34.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.616+0000 7f29eaffd640 1 --2- 172.21.15.133:0/7359724 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec135150 0x7f29ec137570 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704ddf014fa1f660 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.616+0000 7f29ea7fc640 1 -- 172.21.15.133:0/7359724 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f29e0009d90 con 0x7f29ec135150 2024-08-16T02:36:34.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.616+0000 7f29ea7fc640 1 -- 172.21.15.133:0/7359724 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f29e000e230 con 0x7f29ec135150 2024-08-16T02:36:34.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.616+0000 7f29ea7fc640 1 -- 172.21.15.133:0/7359724 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f29e000e4d0 con 0x7f29ec135150 2024-08-16T02:36:34.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.617+0000 7f29f202d640 1 -- 172.21.15.133:0/7359724 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec135150 msgr2=0x7f29ec137570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.617+0000 7f29f202d640 1 --2- 172.21.15.133:0/7359724 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec135150 0x7f29ec137570 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.617+0000 7f29f202d640 1 -- 172.21.15.133:0/7359724 shutdown_connections 2024-08-16T02:36:34.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.617+0000 7f29f202d640 1 --2- 172.21.15.133:0/7359724 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec135150 0x7f29ec137570 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.617+0000 7f29f202d640 1 -- 172.21.15.133:0/7359724 >> 172.21.15.133:0/7359724 conn(0x7f29ec09f1d0 msgr2=0x7f29ec09f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.617+0000 7f29f202d640 1 -- 172.21.15.133:0/7359724 shutdown_connections 2024-08-16T02:36:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.617+0000 7f29f202d640 1 -- 172.21.15.133:0/7359724 wait complete. 2024-08-16T02:36:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.617+0000 7f29f202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.618+0000 7f29f202d640 1 Processor -- start 2024-08-16T02:36:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.619+0000 7f29f202d640 1 -- start start 2024-08-16T02:36:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29f202d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec1322e0 0x7f29ec120bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29f202d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec130fa0 con 0x7f29ec1322e0 2024-08-16T02:36:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29eb7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec1322e0 0x7f29ec120bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29eb7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec1322e0 0x7f29ec120bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56088/0 (socket says 172.21.15.133:56088) 2024-08-16T02:36:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29eb7fe640 1 -- 172.21.15.133:0/1817329319 learned_addr learned my addr 172.21.15.133:0/1817329319 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29eb7fe640 1 -- 172.21.15.133:0/1817329319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec122050 con 0x7f29ec1322e0 2024-08-16T02:36:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29eb7fe640 1 --2- 172.21.15.133:0/1817329319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec1322e0 0x7f29ec120bb0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29d3fff640 1 -- 172.21.15.133:0/1817329319 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f29dc0070b0 con 0x7f29ec1322e0 2024-08-16T02:36:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29d3fff640 1 -- 172.21.15.133:0/1817329319 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f29dc007210 con 0x7f29ec1322e0 2024-08-16T02:36:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec1225e0 con 0x7f29ec1322e0 2024-08-16T02:36:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29d3fff640 1 -- 172.21.15.133:0/1817329319 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f29dc0074b0 con 0x7f29ec1322e0 2024-08-16T02:36:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.620+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec122e90 con 0x7f29ec1322e0 2024-08-16T02:36:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.621+0000 7f29d3fff640 1 -- 172.21.15.133:0/1817329319 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f29dc007610 con 0x7f29ec1322e0 2024-08-16T02:36:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.621+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ec1326f0 con 0x7f29ec1322e0 2024-08-16T02:36:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.622+0000 7f29d3fff640 1 --2- 172.21.15.133:0/1817329319 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f29d803dd10 0x7f29d8040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.622+0000 7f29d3fff640 1 -- 172.21.15.133:0/1817329319 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f29dc04f0d0 con 0x7f29ec1322e0 2024-08-16T02:36:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.622+0000 7f29eaffd640 1 --2- 172.21.15.133:0/1817329319 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f29d803dd10 0x7f29d8040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.623+0000 7f29eaffd640 1 --2- 172.21.15.133:0/1817329319 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f29d803dd10 0x7f29d8040220 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.627+0000 7f29d3fff640 1 -- 172.21.15.133:0/1817329319 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f29dc0090b0 con 0x7f29ec1322e0 2024-08-16T02:36:34.813 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.812+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29ec121ee0 con 0x7f29d803dd10 2024-08-16T02:36:34.815 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.814+0000 7f29d3fff640 1 -- 172.21.15.133:0/1817329319 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31201 (crc 0 0 0) 0x7f29ec121ee0 con 0x7f29d803dd10 2024-08-16T02:36:34.815 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:34.815 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.816+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f29d803dd10 msgr2=0x7f29d8040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.816+0000 7f29f202d640 1 --2- 172.21.15.133:0/1817329319 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f29d803dd10 0x7f29d8040220 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.816+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec1322e0 msgr2=0x7f29ec120bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.816+0000 7f29f202d640 1 --2- 172.21.15.133:0/1817329319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec1322e0 0x7f29ec120bb0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.817+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 shutdown_connections 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.817+0000 7f29f202d640 1 --2- 172.21.15.133:0/1817329319 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f29d803dd10 0x7f29d8040220 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.817+0000 7f29f202d640 1 --2- 172.21.15.133:0/1817329319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29ec1322e0 0x7f29ec120bb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.817+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 >> 172.21.15.133:0/1817329319 conn(0x7f29ec09f1d0 msgr2=0x7f29ec09f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.817+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 shutdown_connections 2024-08-16T02:36:34.818 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.817+0000 7f29f202d640 1 -- 172.21.15.133:0/1817329319 wait complete. 2024-08-16T02:36:34.828 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-08-16T02:36:34.006686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3664,"kb_used_data":3664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209187248,"statfs":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.485710+0000","last_change":"2024-08-16T02:36:32.491674+0000","last_active":"2024-08-16T02:36:32.485710+0000","last_peered":"2024-08-16T02:36:32.485710+0000","last_clean":"2024-08-16T02:36:32.485710+0000","last_became_active":"2024-08-16T02:36:30.530297+0000","last_became_peered":"2024-08-16T02:36:30.530297+0000","last_unstale":"2024-08-16T02:36:32.485710+0000","last_undegraded":"2024-08-16T02:36:32.485710+0000","last_fullsized":"2024-08-16T02:36:32.485710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.489990+0000","last_change":"2024-08-16T02:36:32.493152+0000","last_active":"2024-08-16T02:36:32.489990+0000","last_peered":"2024-08-16T02:36:32.489990+0000","last_clean":"2024-08-16T02:36:32.489990+0000","last_became_active":"2024-08-16T02:36:30.537306+0000","last_became_peered":"2024-08-16T02:36:30.537306+0000","last_unstale":"2024-08-16T02:36:32.489990+0000","last_undegraded":"2024-08-16T02:36:32.489990+0000","last_fullsized":"2024-08-16T02:36:32.489990+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.487256+0000","last_change":"2024-08-16T02:36:32.491067+0000","last_active":"2024-08-16T02:36:32.487256+0000","last_peered":"2024-08-16T02:36:32.487256+0000","last_clean":"2024-08-16T02:36:32.487256+0000","last_became_active":"2024-08-16T02:36:30.537547+0000","last_became_peered":"2024-08-16T02:36:30.537547+0000","last_unstale":"2024-08-16T02:36:32.487256+0000","last_undegraded":"2024-08-16T02:36:32.487256+0000","last_fullsized":"2024-08-16T02:36:32.487256+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.484654+0000","last_change":"2024-08-16T02:36:32.489861+0000","last_active":"2024-08-16T02:36:32.484654+0000","last_peered":"2024-08-16T02:36:32.484654+0000","last_clean":"2024-08-16T02:36:32.484654+0000","last_became_active":"2024-08-16T02:36:30.521009+0000","last_became_peered":"2024-08-16T02:36:30.521009+0000","last_unstale":"2024-08-16T02:36:32.484654+0000","last_undegraded":"2024-08-16T02:36:32.484654+0000","last_fullsized":"2024-08-16T02:36:32.484654+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"14'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.489707+0000","last_change":"2024-08-16T02:36:32.489707+0000","last_active":"2024-08-16T02:36:32.489707+0000","last_peered":"2024-08-16T02:36:32.489707+0000","last_clean":"2024-08-16T02:36:32.489707+0000","last_became_active":"2024-08-16T02:36:30.539438+0000","last_became_peered":"2024-08-16T02:36:30.539438+0000","last_unstale":"2024-08-16T02:36:32.489707+0000","last_undegraded":"2024-08-16T02:36:32.489707+0000","last_fullsized":"2024-08-16T02:36:32.489707+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.484984+0000","last_change":"2024-08-16T02:36:32.491302+0000","last_active":"2024-08-16T02:36:32.484984+0000","last_peered":"2024-08-16T02:36:32.484984+0000","last_clean":"2024-08-16T02:36:32.484984+0000","last_became_active":"2024-08-16T02:36:30.521005+0000","last_became_peered":"2024-08-16T02:36:30.521005+0000","last_unstale":"2024-08-16T02:36:32.484984+0000","last_undegraded":"2024-08-16T02:36:32.484984+0000","last_fullsized":"2024-08-16T02:36:32.484984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.486362+0000","last_change":"2024-08-16T02:36:32.490191+0000","last_active":"2024-08-16T02:36:32.486362+0000","last_peered":"2024-08-16T02:36:32.486362+0000","last_clean":"2024-08-16T02:36:32.486362+0000","last_became_active":"2024-08-16T02:36:30.548723+0000","last_became_peered":"2024-08-16T02:36:30.548723+0000","last_unstale":"2024-08-16T02:36:32.486362+0000","last_undegraded":"2024-08-16T02:36:32.486362+0000","last_fullsized":"2024-08-16T02:36:32.486362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.491907+0000","last_change":"2024-08-16T02:36:32.495617+0000","last_active":"2024-08-16T02:36:32.491907+0000","last_peered":"2024-08-16T02:36:32.491907+0000","last_clean":"2024-08-16T02:36:32.491907+0000","last_became_active":"2024-08-16T02:36:30.528787+0000","last_became_peered":"2024-08-16T02:36:30.528787+0000","last_unstale":"2024-08-16T02:36:32.491907+0000","last_undegraded":"2024-08-16T02:36:32.491907+0000","last_fullsized":"2024-08-16T02:36:32.491907+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.484406+0000","last_change":"2024-08-16T02:36:28.507195+0000","last_active":"2024-08-16T02:36:32.484406+0000","last_peered":"2024-08-16T02:36:32.484406+0000","last_clean":"2024-08-16T02:36:32.484406+0000","last_became_active":"2024-08-16T02:36:28.489045+0000","last_became_peered":"2024-08-16T02:36:28.489045+0000","last_unstale":"2024-08-16T02:36:32.484406+0000","last_undegraded":"2024-08-16T02:36:32.484406+0000","last_fullsized":"2024-08-16T02:36:32.484406+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_clean_scrub_stamp":"2024-08-16T02:36:27.432441+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1172,"kb_used_data":1172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296556,"statfs":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":532,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297196,"statfs":{"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":528,"kb_used_data":528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297200,"statfs":{"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1432,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296296,"statfs":{"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:36:34.830 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:36:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.934+0000 7fe65f592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.939 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.938+0000 7fe65f592640 1 Processor -- start 2024-08-16T02:36:34.939 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.939+0000 7fe65f592640 1 -- start start 2024-08-16T02:36:34.941 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65f592640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe65809cb30 0x7fe65809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.941 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65f592640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6580a0260 con 0x7fe65809d480 2024-08-16T02:36:34.941 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65f592640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7fe658070900 con 0x7fe65809cb30 2024-08-16T02:36:34.941 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65d32e640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe65809cb30 0x7fe65809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65cb2d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fe65809d480 0x7fe6581322f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34364/0 (socket says 172.21.15.133:34364) 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65d32e640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe65809cb30 0x7fe65809cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56102/0 (socket says 172.21.15.133:56102) 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65cb2d640 1 -- 172.21.15.133:0/4191464134 learned_addr learned my addr 172.21.15.133:0/4191464134 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65d32e640 1 -- 172.21.15.133:0/4191464134 >> v1:172.21.15.133:6789/0 conn(0x7fe65809d480 legacy=0x7fe6581322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65d32e640 1 -- 172.21.15.133:0/4191464134 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65806f710 con 0x7fe65809cb30 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.940+0000 7fe65d32e640 1 --2- 172.21.15.133:0/4191464134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe65809cf40 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a526a27802511548 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe64ffff640 1 -- 172.21.15.133:0/4191464134 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe654009d90 con 0x7fe65809cb30 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe64ffff640 1 -- 172.21.15.133:0/4191464134 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe65400e230 con 0x7fe65809cb30 2024-08-16T02:36:34.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe64ffff640 1 -- 172.21.15.133:0/4191464134 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe65400e4d0 con 0x7fe65809cb30 2024-08-16T02:36:34.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe65f592640 1 -- 172.21.15.133:0/4191464134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 msgr2=0x7fe65809cf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe65f592640 1 --2- 172.21.15.133:0/4191464134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe65809cf40 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe65f592640 1 -- 172.21.15.133:0/4191464134 shutdown_connections 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe65f592640 1 --2- 172.21.15.133:0/4191464134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe65809cf40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe65f592640 1 -- 172.21.15.133:0/4191464134 >> 172.21.15.133:0/4191464134 conn(0x7fe6580a09c0 msgr2=0x7fe6580a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe65f592640 1 -- 172.21.15.133:0/4191464134 shutdown_connections 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.941+0000 7fe65f592640 1 -- 172.21.15.133:0/4191464134 wait complete. 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.942+0000 7fe65f592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.943+0000 7fe65f592640 1 Processor -- start 2024-08-16T02:36:34.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.943+0000 7fe65f592640 1 -- start start 2024-08-16T02:36:34.946 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.944+0000 7fe65f592640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe6581c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.946 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.944+0000 7fe65f592640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7fe6581333a0 con 0x7fe65809cb30 2024-08-16T02:36:34.946 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.944+0000 7fe65d32e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe6581c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.946 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.944+0000 7fe65d32e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe6581c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56104/0 (socket says 172.21.15.133:56104) 2024-08-16T02:36:34.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.944+0000 7fe65d32e640 1 -- 172.21.15.133:0/4273843391 learned_addr learned my addr 172.21.15.133:0/4273843391 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:34.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.944+0000 7fe65d32e640 1 -- 172.21.15.133:0/4273843391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe658123320 con 0x7fe65809cb30 2024-08-16T02:36:34.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.944+0000 7fe65d32e640 1 --2- 172.21.15.133:0/4273843391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe6581c7fc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.945+0000 7fe64dffb640 1 -- 172.21.15.133:0/4273843391 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe654013530 con 0x7fe65809cb30 2024-08-16T02:36:34.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.945+0000 7fe64dffb640 1 -- 172.21.15.133:0/4273843391 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe654013690 con 0x7fe65809cb30 2024-08-16T02:36:34.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.945+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6581c7c00 con 0x7fe65809cb30 2024-08-16T02:36:34.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.945+0000 7fe64dffb640 1 -- 172.21.15.133:0/4273843391 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe654013930 con 0x7fe65809cb30 2024-08-16T02:36:34.947 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.945+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe658123940 con 0x7fe65809cb30 2024-08-16T02:36:34.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.946+0000 7fe64dffb640 1 -- 172.21.15.133:0/4273843391 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fe654013b30 con 0x7fe65809cb30 2024-08-16T02:36:34.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.946+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe658124d90 con 0x7fe65809cb30 2024-08-16T02:36:34.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.946+0000 7fe64dffb640 1 --2- 172.21.15.133:0/4273843391 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe63c03dcc0 0x7fe63c0401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:34.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.946+0000 7fe64dffb640 1 -- 172.21.15.133:0/4273843391 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fe654018040 con 0x7fe65809cb30 2024-08-16T02:36:34.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.947+0000 7fe65cb2d640 1 --2- 172.21.15.133:0/4273843391 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe63c03dcc0 0x7fe63c0401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:34.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.947+0000 7fe65cb2d640 1 --2- 172.21.15.133:0/4273843391 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe63c03dcc0 0x7fe63c0401d0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:34.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:34.951+0000 7fe64dffb640 1 -- 172.21.15.133:0/4273843391 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fe654011380 con 0x7fe65809cb30 2024-08-16T02:36:35.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.132+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe65809ff50 con 0x7fe63c03dcc0 2024-08-16T02:36:35.134 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.134+0000 7fe64dffb640 1 -- 172.21.15.133:0/4273843391 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31201 (crc 0 0 0) 0x7fe6480058d0 con 0x7fe63c03dcc0 2024-08-16T02:36:35.135 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:35.135 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-08-16T02:36:35.137 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe63c03dcc0 msgr2=0x7fe63c0401d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:35.137 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 --2- 172.21.15.133:0/4273843391 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe63c03dcc0 0x7fe63c0401d0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.137 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 msgr2=0x7fe6581c7fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:35.137 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 --2- 172.21.15.133:0/4273843391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe6581c7fc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.138 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 shutdown_connections 2024-08-16T02:36:35.138 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 --2- 172.21.15.133:0/4273843391 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe63c03dcc0 0x7fe63c0401d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.138 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 --2- 172.21.15.133:0/4273843391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe65809cb30 0x7fe6581c7fc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.138 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 >> 172.21.15.133:0/4273843391 conn(0x7fe6580a09c0 msgr2=0x7fe658072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:35.138 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 shutdown_connections 2024-08-16T02:36:35.138 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.136+0000 7fe65f592640 1 -- 172.21.15.133:0/4273843391 wait complete. 2024-08-16T02:36:35.148 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-08-16T02:36:34.006686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3664,"kb_used_data":3664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209187248,"statfs":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.485710+0000","last_change":"2024-08-16T02:36:32.491674+0000","last_active":"2024-08-16T02:36:32.485710+0000","last_peered":"2024-08-16T02:36:32.485710+0000","last_clean":"2024-08-16T02:36:32.485710+0000","last_became_active":"2024-08-16T02:36:30.530297+0000","last_became_peered":"2024-08-16T02:36:30.530297+0000","last_unstale":"2024-08-16T02:36:32.485710+0000","last_undegraded":"2024-08-16T02:36:32.485710+0000","last_fullsized":"2024-08-16T02:36:32.485710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.489990+0000","last_change":"2024-08-16T02:36:32.493152+0000","last_active":"2024-08-16T02:36:32.489990+0000","last_peered":"2024-08-16T02:36:32.489990+0000","last_clean":"2024-08-16T02:36:32.489990+0000","last_became_active":"2024-08-16T02:36:30.537306+0000","last_became_peered":"2024-08-16T02:36:30.537306+0000","last_unstale":"2024-08-16T02:36:32.489990+0000","last_undegraded":"2024-08-16T02:36:32.489990+0000","last_fullsized":"2024-08-16T02:36:32.489990+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.487256+0000","last_change":"2024-08-16T02:36:32.491067+0000","last_active":"2024-08-16T02:36:32.487256+0000","last_peered":"2024-08-16T02:36:32.487256+0000","last_clean":"2024-08-16T02:36:32.487256+0000","last_became_active":"2024-08-16T02:36:30.537547+0000","last_became_peered":"2024-08-16T02:36:30.537547+0000","last_unstale":"2024-08-16T02:36:32.487256+0000","last_undegraded":"2024-08-16T02:36:32.487256+0000","last_fullsized":"2024-08-16T02:36:32.487256+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.484654+0000","last_change":"2024-08-16T02:36:32.489861+0000","last_active":"2024-08-16T02:36:32.484654+0000","last_peered":"2024-08-16T02:36:32.484654+0000","last_clean":"2024-08-16T02:36:32.484654+0000","last_became_active":"2024-08-16T02:36:30.521009+0000","last_became_peered":"2024-08-16T02:36:30.521009+0000","last_unstale":"2024-08-16T02:36:32.484654+0000","last_undegraded":"2024-08-16T02:36:32.484654+0000","last_fullsized":"2024-08-16T02:36:32.484654+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"14'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.489707+0000","last_change":"2024-08-16T02:36:32.489707+0000","last_active":"2024-08-16T02:36:32.489707+0000","last_peered":"2024-08-16T02:36:32.489707+0000","last_clean":"2024-08-16T02:36:32.489707+0000","last_became_active":"2024-08-16T02:36:30.539438+0000","last_became_peered":"2024-08-16T02:36:30.539438+0000","last_unstale":"2024-08-16T02:36:32.489707+0000","last_undegraded":"2024-08-16T02:36:32.489707+0000","last_fullsized":"2024-08-16T02:36:32.489707+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.484984+0000","last_change":"2024-08-16T02:36:32.491302+0000","last_active":"2024-08-16T02:36:32.484984+0000","last_peered":"2024-08-16T02:36:32.484984+0000","last_clean":"2024-08-16T02:36:32.484984+0000","last_became_active":"2024-08-16T02:36:30.521005+0000","last_became_peered":"2024-08-16T02:36:30.521005+0000","last_unstale":"2024-08-16T02:36:32.484984+0000","last_undegraded":"2024-08-16T02:36:32.484984+0000","last_fullsized":"2024-08-16T02:36:32.484984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.486362+0000","last_change":"2024-08-16T02:36:32.490191+0000","last_active":"2024-08-16T02:36:32.486362+0000","last_peered":"2024-08-16T02:36:32.486362+0000","last_clean":"2024-08-16T02:36:32.486362+0000","last_became_active":"2024-08-16T02:36:30.548723+0000","last_became_peered":"2024-08-16T02:36:30.548723+0000","last_unstale":"2024-08-16T02:36:32.486362+0000","last_undegraded":"2024-08-16T02:36:32.486362+0000","last_fullsized":"2024-08-16T02:36:32.486362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.491907+0000","last_change":"2024-08-16T02:36:32.495617+0000","last_active":"2024-08-16T02:36:32.491907+0000","last_peered":"2024-08-16T02:36:32.491907+0000","last_clean":"2024-08-16T02:36:32.491907+0000","last_became_active":"2024-08-16T02:36:30.528787+0000","last_became_peered":"2024-08-16T02:36:30.528787+0000","last_unstale":"2024-08-16T02:36:32.491907+0000","last_undegraded":"2024-08-16T02:36:32.491907+0000","last_fullsized":"2024-08-16T02:36:32.491907+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:36:32.484406+0000","last_change":"2024-08-16T02:36:28.507195+0000","last_active":"2024-08-16T02:36:32.484406+0000","last_peered":"2024-08-16T02:36:32.484406+0000","last_clean":"2024-08-16T02:36:32.484406+0000","last_became_active":"2024-08-16T02:36:28.489045+0000","last_became_peered":"2024-08-16T02:36:28.489045+0000","last_unstale":"2024-08-16T02:36:32.484406+0000","last_undegraded":"2024-08-16T02:36:32.484406+0000","last_fullsized":"2024-08-16T02:36:32.484406+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_clean_scrub_stamp":"2024-08-16T02:36:27.432441+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1172,"kb_used_data":1172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296556,"statfs":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":532,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297196,"statfs":{"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":528,"kb_used_data":528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297200,"statfs":{"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1432,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296296,"statfs":{"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:36:35.149 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-16T02:36:35.149 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-16T02:36:35.149 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-16T02:36:35.149 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-16T02:36:35.257 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.253+0000 7fe90d224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:35.257 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.257+0000 7fe90d224640 1 Processor -- start 2024-08-16T02:36:35.257 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.257+0000 7fe90d224640 1 -- start start 2024-08-16T02:36:35.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe90d224640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe9081322e0 0x7fe9081326f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:35.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe90d224640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe90809e6b0 con 0x7fe90809ba20 2024-08-16T02:36:35.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe90d224640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7fe90809e3a0 con 0x7fe9081322e0 2024-08-16T02:36:35.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe906d9d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe9081322e0 0x7fe9081326f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe90659c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fe90809ba20 0x7fe90809be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:34380/0 (socket says 172.21.15.133:34380) 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe90659c640 1 -- 172.21.15.133:0/967376183 learned_addr learned my addr 172.21.15.133:0/967376183 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe906d9d640 1 -- 172.21.15.133:0/967376183 >> v1:172.21.15.133:6789/0 conn(0x7fe90809ba20 legacy=0x7fe90809be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe906d9d640 1 -- 172.21.15.133:0/967376183 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90809cf70 con 0x7fe9081322e0 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe905d9b640 1 -- 172.21.15.133:0/967376183 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311047502 0 0) 0x7fe90809e6b0 con 0x7fe90809ba20 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe906d9d640 1 --2- 172.21.15.133:0/967376183 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9081322e0 0x7fe9081326f0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44248069e044677d server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.258+0000 7fe905d9b640 1 -- 172.21.15.133:0/967376183 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe8f8009d90 con 0x7fe9081322e0 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe905d9b640 1 -- 172.21.15.133:0/967376183 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe8f800e230 con 0x7fe9081322e0 2024-08-16T02:36:35.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe905d9b640 1 -- 172.21.15.133:0/967376183 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe8f800e4d0 con 0x7fe9081322e0 2024-08-16T02:36:35.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe90d224640 1 -- 172.21.15.133:0/967376183 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9081322e0 msgr2=0x7fe9081326f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:35.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe90d224640 1 --2- 172.21.15.133:0/967376183 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9081322e0 0x7fe9081326f0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe90d224640 1 -- 172.21.15.133:0/967376183 shutdown_connections 2024-08-16T02:36:35.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe90d224640 1 --2- 172.21.15.133:0/967376183 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9081322e0 0x7fe9081326f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe90d224640 1 -- 172.21.15.133:0/967376183 >> 172.21.15.133:0/967376183 conn(0x7fe90809f1d0 msgr2=0x7fe90809f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:35.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe90d224640 1 -- 172.21.15.133:0/967376183 shutdown_connections 2024-08-16T02:36:35.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe90d224640 1 -- 172.21.15.133:0/967376183 wait complete. 2024-08-16T02:36:35.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.259+0000 7fe90d224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:35.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.261+0000 7fe90d224640 1 Processor -- start 2024-08-16T02:36:35.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.261+0000 7fe90d224640 1 -- start start 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.262+0000 7fe90d224640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe90809ba20 0x7fe90811cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.262+0000 7fe90d224640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7fe908120c30 con 0x7fe90809ba20 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.262+0000 7fe906d9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe90809ba20 0x7fe90811cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.262+0000 7fe906d9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe90809ba20 0x7fe90811cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56126/0 (socket says 172.21.15.133:56126) 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.262+0000 7fe906d9d640 1 -- 172.21.15.133:0/1897222788 learned_addr learned my addr 172.21.15.133:0/1897222788 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.263+0000 7fe906d9d640 1 -- 172.21.15.133:0/1897222788 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe908120900 con 0x7fe90809ba20 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.263+0000 7fe906d9d640 1 --2- 172.21.15.133:0/1897222788 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe90809ba20 0x7fe90811cc70 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.263+0000 7fe8ef7fe640 1 -- 172.21.15.133:0/1897222788 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe8f8013530 con 0x7fe90809ba20 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.263+0000 7fe8ef7fe640 1 -- 172.21.15.133:0/1897222788 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe8f8013690 con 0x7fe90809ba20 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.263+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe908131250 con 0x7fe90809ba20 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.263+0000 7fe8ef7fe640 1 -- 172.21.15.133:0/1897222788 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe8f8013930 con 0x7fe90809ba20 2024-08-16T02:36:35.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.263+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe908120510 con 0x7fe90809ba20 2024-08-16T02:36:35.265 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.264+0000 7fe8ef7fe640 1 -- 172.21.15.133:0/1897222788 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fe8f8013ae0 con 0x7fe90809ba20 2024-08-16T02:36:35.265 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.264+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9080008d0 con 0x7fe90809ba20 2024-08-16T02:36:35.265 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.264+0000 7fe8ef7fe640 1 --2- 172.21.15.133:0/1897222788 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe8f403dd10 0x7fe8f4040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:35.266 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.264+0000 7fe8ef7fe640 1 -- 172.21.15.133:0/1897222788 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fe8f8018040 con 0x7fe90809ba20 2024-08-16T02:36:35.266 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.264+0000 7fe90659c640 1 --2- 172.21.15.133:0/1897222788 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe8f403dd10 0x7fe8f4040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:35.266 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.265+0000 7fe90659c640 1 --2- 172.21.15.133:0/1897222788 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe8f403dd10 0x7fe8f4040220 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:35.270 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.269+0000 7fe8ef7fe640 1 -- 172.21.15.133:0/1897222788 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fe8f8018c20 con 0x7fe90809ba20 2024-08-16T02:36:35.496 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.495+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe90811f4d0 con 0x7fe90809ba20 2024-08-16T02:36:35.496 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.495+0000 7fe8ef7fe640 1 -- 172.21.15.133:0/1897222788 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (crc 0 0 0) 0x7fe8f8018a40 con 0x7fe90809ba20 2024-08-16T02:36:35.496 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:36:35.496 INFO:teuthology.orchestra.run.smithi133.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-16T02:36:35.498 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe8f403dd10 msgr2=0x7fe8f4040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:35.498 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 --2- 172.21.15.133:0/1897222788 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe8f403dd10 0x7fe8f4040220 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe90809ba20 msgr2=0x7fe90811cc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:35.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 --2- 172.21.15.133:0/1897222788 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe90809ba20 0x7fe90811cc70 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 shutdown_connections 2024-08-16T02:36:35.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 --2- 172.21.15.133:0/1897222788 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fe8f403dd10 0x7fe8f4040220 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 --2- 172.21.15.133:0/1897222788 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe90809ba20 0x7fe90811cc70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:35.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 >> 172.21.15.133:0/1897222788 conn(0x7fe90809f1d0 msgr2=0x7fe9081b9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:35.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 shutdown_connections 2024-08-16T02:36:35.499 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:36:35.497+0000 7fe90d224640 1 -- 172.21.15.133:0/1897222788 wait complete. 2024-08-16T02:36:35.509 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-16T02:36:35.510 INFO:teuthology.run_tasks:Running task workunit... 2024-08-16T02:36:35.524 INFO:tasks.workunit:Pulling workunits from ref 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:36:35.524 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-16T02:36:35.524 DEBUG:teuthology.orchestra.run.smithi133:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-16T02:36:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:36:35.540 INFO:teuthology.orchestra.run.smithi133.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-16T02:36:35.540 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-16T02:36:35.595 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-16T02:36:35.595 DEBUG:teuthology.orchestra.run.smithi133:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-16T02:36:35.649 INFO:tasks.workunit:timeout=3h 2024-08-16T02:36:35.650 INFO:tasks.workunit:cleanup=True 2024-08-16T02:36:35.650 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:36:35.705 INFO:tasks.workunit.client.0.smithi133.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-16T02:37:33.484 INFO:tasks.workunit.client.0.smithi133.stderr:Note: switching to '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'. 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr:state without impacting any branches by switching back to a branch. 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr: git switch -c 2024-08-16T02:37:33.485 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-08-16T02:37:33.486 INFO:tasks.workunit.client.0.smithi133.stderr:Or undo this operation with: 2024-08-16T02:37:33.486 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-08-16T02:37:33.486 INFO:tasks.workunit.client.0.smithi133.stderr: git switch - 2024-08-16T02:37:33.486 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-08-16T02:37:33.486 INFO:tasks.workunit.client.0.smithi133.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-16T02:37:33.486 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-08-16T02:37:33.486 INFO:tasks.workunit.client.0.smithi133.stderr:HEAD is now at 6f386a3f0b1 Merge branch 'wip-seastore-more-reports' of https://github.com/cyx1231st/ceph into wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:37:33.488 DEBUG:teuthology.orchestra.run.smithi133:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-16T02:37:33.578 INFO:tasks.workunit.client.0.smithi133.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-16T02:37:33.580 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-16T02:37:33.580 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-16T02:37:34.231 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-16T02:37:34.288 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-16T02:37:34.329 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-16T02:37:34.330 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-16T02:37:34.330 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-16T02:37:34.370 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-16T02:37:34.372 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-08-16T02:37:34.372 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-16T02:37:34.389 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-08-16T02:37:34.390 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-08-16T02:37:34.390 DEBUG:teuthology.orchestra.run.smithi133:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson 2024-08-16T02:37:34.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ parallel=1 2024-08-16T02:37:34.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ '[' --crimson = --serial ']' 2024-08-16T02:37:34.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ crimson=0 2024-08-16T02:37:34.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ '[' --crimson = --crimson ']' 2024-08-16T02:37:34.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ parallel=0 2024-08-16T02:37:34.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ crimson=1 2024-08-16T02:37:34.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ color= 2024-08-16T02:37:34.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ '[' -t 1 ']' 2024-08-16T02:37:34.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-08-16T02:37:34.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-16T02:37:34.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-16T02:37:34.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ declare -A pids 2024-08-16T02:37:34.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-16T02:37:34.451 INFO:tasks.workunit.client.0.smithi133.stderr:+ '[' 0 -eq 1 ']' 2024-08-16T02:37:34.451 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph_test_rados_api_aio 2024-08-16T02:37:34.467 INFO:tasks.workunit.client.0.smithi133.stdout:Running main() from gmock_main.cc 2024-08-16T02:37:34.468 INFO:tasks.workunit.client.0.smithi133.stdout:[==========] Running 42 tests from 2 test suites. 2024-08-16T02:37:34.468 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] Global test environment set-up. 2024-08-16T02:37:34.468 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 26 tests from LibRadosAio 2024-08-16T02:37:34.468 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.TooBig 2024-08-16T02:37:36.801 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.TooBig (2334 ms) 2024-08-16T02:37:36.802 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.SimpleWrite 2024-08-16T02:37:39.821 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.SimpleWrite (3020 ms) 2024-08-16T02:37:39.821 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.WaitForSafe 2024-08-16T02:37:42.834 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.WaitForSafe (3013 ms) 2024-08-16T02:37:42.834 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTrip 2024-08-16T02:37:45.860 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTrip (3026 ms) 2024-08-16T02:37:45.860 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTrip2 2024-08-16T02:37:48.871 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTrip2 (3011 ms) 2024-08-16T02:37:48.871 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTrip3 2024-08-16T02:37:51.892 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTrip3 (3021 ms) 2024-08-16T02:37:51.892 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripAppend 2024-08-16T02:37:54.906 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripAppend (3015 ms) 2024-08-16T02:37:54.907 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RemoveTest 2024-08-16T02:37:57.935 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RemoveTest (3028 ms) 2024-08-16T02:37:57.935 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.XattrsRoundTrip 2024-08-16T02:38:00.947 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.XattrsRoundTrip (3012 ms) 2024-08-16T02:38:00.947 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RmXattr 2024-08-16T02:38:03.970 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RmXattr (3024 ms) 2024-08-16T02:38:03.971 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.XattrIter 2024-08-16T02:38:06.978 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.XattrIter (3007 ms) 2024-08-16T02:38:06.978 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.IsComplete 2024-08-16T02:38:09.996 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.IsComplete (3018 ms) 2024-08-16T02:38:09.996 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.IsSafe 2024-08-16T02:38:13.008 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.IsSafe (3012 ms) 2024-08-16T02:38:13.009 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.ReturnValue 2024-08-16T02:38:16.030 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.ReturnValue (3022 ms) 2024-08-16T02:38:16.030 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.Flush 2024-08-16T02:38:19.039 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.Flush (3009 ms) 2024-08-16T02:38:19.039 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.FlushAsync 2024-08-16T02:38:22.054 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.FlushAsync (3015 ms) 2024-08-16T02:38:22.055 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripWriteFull 2024-08-16T02:38:25.071 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripWriteFull (3017 ms) 2024-08-16T02:38:25.071 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripWriteSame 2024-08-16T02:38:28.088 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripWriteSame (3017 ms) 2024-08-16T02:38:28.088 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.SimpleStat 2024-08-16T02:38:30.792 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.SimpleStat (2704 ms) 2024-08-16T02:38:30.792 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.OperateMtime 2024-08-16T02:38:33.806 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.OperateMtime (3014 ms) 2024-08-16T02:38:33.807 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.Operate2Mtime 2024-08-16T02:38:36.817 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.Operate2Mtime (3011 ms) 2024-08-16T02:38:36.817 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.SimpleStatNS 2024-08-16T02:38:39.825 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.SimpleStatNS (3008 ms) 2024-08-16T02:38:39.825 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.StatRemove 2024-08-16T02:38:42.836 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.StatRemove (3010 ms) 2024-08-16T02:38:42.836 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.ExecuteClass 2024-08-16T02:38:45.860 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.ExecuteClass (3025 ms) 2024-08-16T02:38:45.860 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.MultiWrite 2024-08-16T02:38:48.794 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.MultiWrite (2934 ms) 2024-08-16T02:38:48.794 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.AioUnlock 2024-08-16T02:38:51.807 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.AioUnlock (3012 ms) 2024-08-16T02:38:51.807 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 26 tests from LibRadosAio (77339 ms total) 2024-08-16T02:38:51.807 INFO:tasks.workunit.client.0.smithi133.stdout: 2024-08-16T02:38:51.807 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 16 tests from LibRadosAioEC 2024-08-16T02:38:51.807 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.SimpleWrite 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1100: Skipped 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWrite (1 ms) 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.WaitForComplete 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1132: Skipped 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.WaitForComplete (0 ms) 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTrip 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1149: Skipped 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.808 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip (0 ms) 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTrip2 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1182: Skipped 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip2 (0 ms) 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripAppend 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1215: Skipped 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppend (0 ms) 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.IsComplete 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1286: Skipped 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.IsComplete (0 ms) 2024-08-16T02:38:51.809 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.IsSafe 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1326: Skipped 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.IsSafe (0 ms) 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.ReturnValue 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1366: Skipped 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValue (0 ms) 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.Flush 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1385: Skipped 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.Flush (0 ms) 2024-08-16T02:38:51.810 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.FlushAsync 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1415: Skipped 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsync (0 ms) 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1454: Skipped 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFull (0 ms) 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.SimpleStat 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1498: Skipped 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStat (0 ms) 2024-08-16T02:38:51.811 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.SimpleStatNS 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1532: Skipped 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatNS (0 ms) 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.StatRemove 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1593: Skipped 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.StatRemove (0 ms) 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.ExecuteClass 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1649: Skipped 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClass (0 ms) 2024-08-16T02:38:51.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.MultiWrite 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1681: Skipped 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.MultiWrite (0 ms) 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 16 tests from LibRadosAioEC (1 ms total) 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout: 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] Global test environment tear-down 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[==========] 42 tests from 2 test suites ran. (77340 ms total) 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[ PASSED ] 26 tests. 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] 16 tests, listed below: 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWrite 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.WaitForComplete 2024-08-16T02:38:51.813 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip2 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppend 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.IsComplete 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.IsSafe 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValue 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.Flush 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsync 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFull 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStat 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatNS 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.StatRemove 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClass 2024-08-16T02:38:51.814 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.MultiWrite 2024-08-16T02:38:51.815 INFO:tasks.workunit.client.0.smithi133.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-16T02:38:51.815 INFO:tasks.workunit.client.0.smithi133.stderr:+ '[' 0 -eq 1 ']' 2024-08-16T02:38:51.815 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph_test_rados_api_aio_pp 2024-08-16T02:38:51.831 INFO:tasks.workunit.client.0.smithi133.stdout:Running main() from gmock_main.cc 2024-08-16T02:38:51.831 INFO:tasks.workunit.client.0.smithi133.stdout:[==========] Running 57 tests from 4 test suites. 2024-08-16T02:38:51.831 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] Global test environment set-up. 2024-08-16T02:38:51.832 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 32 tests from LibRadosAio 2024-08-16T02:38:51.832 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.TooBigPP 2024-08-16T02:38:54.815 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.TooBigPP (2984 ms) 2024-08-16T02:38:54.815 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.PoolQuotaPP 2024-08-16T02:39:10.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.PoolQuotaPP (15997 ms) 2024-08-16T02:39:10.813 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.SimpleWritePP 2024-08-16T02:39:16.841 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.SimpleWritePP (6029 ms) 2024-08-16T02:39:16.841 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.WaitForSafePP 2024-08-16T02:39:19.849 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.WaitForSafePP (3008 ms) 2024-08-16T02:39:19.850 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripPP 2024-08-16T02:39:22.867 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripPP (3018 ms) 2024-08-16T02:39:22.868 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripPP2 2024-08-16T02:39:25.881 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripPP2 (3014 ms) 2024-08-16T02:39:25.881 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripPP3 2024-08-16T02:39:28.804 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripPP3 (2922 ms) 2024-08-16T02:39:28.804 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-08-16T02:39:31.822 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripSparseReadPP (3019 ms) 2024-08-16T02:39:31.823 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.IsCompletePP 2024-08-16T02:39:34.838 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.IsCompletePP (3016 ms) 2024-08-16T02:39:34.838 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.IsSafePP 2024-08-16T02:39:37.859 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.IsSafePP (3021 ms) 2024-08-16T02:39:37.859 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.ReturnValuePP 2024-08-16T02:39:40.865 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.ReturnValuePP (3006 ms) 2024-08-16T02:39:40.866 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.FlushPP 2024-08-16T02:39:43.874 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.FlushPP (3009 ms) 2024-08-16T02:39:43.874 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.FlushAsyncPP 2024-08-16T02:39:46.885 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.FlushAsyncPP (3011 ms) 2024-08-16T02:39:46.885 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-08-16T02:39:49.894 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripWriteFullPP (3010 ms) 2024-08-16T02:39:49.895 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-08-16T02:39:52.903 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripWriteFullPP2 (3009 ms) 2024-08-16T02:39:52.904 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-08-16T02:39:55.927 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripWriteSamePP (3023 ms) 2024-08-16T02:39:55.927 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-08-16T02:39:58.805 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripWriteSamePP2 (2878 ms) 2024-08-16T02:39:58.805 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.SimpleStatPPNS 2024-08-16T02:40:01.823 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.SimpleStatPPNS (3018 ms) 2024-08-16T02:40:01.823 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.SimpleStatPP 2024-08-16T02:40:04.832 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.SimpleStatPP (3009 ms) 2024-08-16T02:40:04.832 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.OperateMtime 2024-08-16T02:40:07.849 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.OperateMtime (3017 ms) 2024-08-16T02:40:07.849 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.OperateMtime2 2024-08-16T02:40:10.851 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.OperateMtime2 (3002 ms) 2024-08-16T02:40:10.851 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.StatRemovePP 2024-08-16T02:40:13.873 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.StatRemovePP (3022 ms) 2024-08-16T02:40:13.873 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.ExecuteClassPP 2024-08-16T02:40:16.889 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.ExecuteClassPP (3016 ms) 2024-08-16T02:40:16.889 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.OmapPP 2024-08-16T02:40:19.904 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.OmapPP (3015 ms) 2024-08-16T02:40:19.904 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.MultiWritePP 2024-08-16T02:40:22.913 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.MultiWritePP (3009 ms) 2024-08-16T02:40:22.913 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.AioUnlockPP 2024-08-16T02:40:25.936 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.AioUnlockPP (3022 ms) 2024-08-16T02:40:25.936 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripAppendPP 2024-08-16T02:40:28.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripAppendPP (2876 ms) 2024-08-16T02:40:28.812 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RacingRemovePP 2024-08-16T02:40:31.835 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RacingRemovePP (3023 ms) 2024-08-16T02:40:31.835 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-08-16T02:40:34.844 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripCmpExtPP (3009 ms) 2024-08-16T02:40:34.844 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-08-16T02:40:37.860 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAio.RoundTripCmpExtPP2 (3016 ms) 2024-08-16T02:40:37.861 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.PoolEIOFlag 2024-08-16T02:40:40.312 INFO:tasks.workunit.client.0.smithi133.stdout:setting pool EIO 2024-08-16T02:40:49.979 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 692 2024-08-16T02:40:50.979 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 613 2024-08-16T02:40:51.979 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 532 2024-08-16T02:40:52.979 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 452 2024-08-16T02:40:53.979 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 373 2024-08-16T02:40:54.979 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 294 2024-08-16T02:40:55.980 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 214 2024-08-16T02:40:56.980 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 135 2024-08-16T02:40:57.980 INFO:tasks.workunit.client.0.smithi133.stdout:waiting for 53 2024-08-16T02:40:58.980 INFO:tasks.workunit.client.0.smithi133.stdout:max_success 1000, min_failed 0 2024-08-16T02:40:58.980 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2421: Failure 2024-08-16T02:40:58.980 INFO:tasks.workunit.client.0.smithi133.stdout:Value of: max_success + 1 == min_failed 2024-08-16T02:40:58.980 INFO:tasks.workunit.client.0.smithi133.stdout: Actual: false 2024-08-16T02:40:58.980 INFO:tasks.workunit.client.0.smithi133.stdout:Expected: true 2024-08-16T02:40:59.830 INFO:tasks.workunit.client.0.smithi133.stdout:[ FAILED ] LibRadosAio.PoolEIOFlag (21970 ms) 2024-08-16T02:40:59.831 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAio.MultiReads 2024-08-16T02:40:59.831 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2427: Skipped 2024-08-16T02:40:59.831 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:40:59.831 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAio.MultiReads (0 ms) 2024-08-16T02:40:59.831 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 32 tests from LibRadosAio (127999 ms total) 2024-08-16T02:40:59.831 INFO:tasks.workunit.client.0.smithi133.stdout: 2024-08-16T02:40:59.831 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 4 tests from LibRadosAioPP 2024-08-16T02:40:59.831 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-08-16T02:41:02.839 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAioPP.ReadIntoBufferlist (3009 ms) 2024-08-16T02:41:02.839 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-08-16T02:41:11.831 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAioPP.XattrsRoundTripPP (8992 ms) 2024-08-16T02:41:11.831 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioPP.RmXattrPP 2024-08-16T02:41:26.896 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAioPP.RmXattrPP (15064 ms) 2024-08-16T02:41:26.896 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioPP.RemoveTestPP 2024-08-16T02:41:29.921 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosAioPP.RemoveTestPP (3026 ms) 2024-08-16T02:41:29.921 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 4 tests from LibRadosAioPP (30091 ms total) 2024-08-16T02:41:29.922 INFO:tasks.workunit.client.0.smithi133.stdout: 2024-08-16T02:41:29.922 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 1 test from LibRadosIoPP 2024-08-16T02:41:29.922 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosIoPP.XattrListPP 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout:[ OK ] LibRadosIoPP.XattrListPP (3018 ms) 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 1 test from LibRadosIoPP (3018 ms total) 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout: 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 20 tests from LibRadosAioEC 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.SimpleWritePP 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1384: Skipped 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWritePP (1 ms) 2024-08-16T02:41:32.941 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.WaitForSafePP 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1419: Skipped 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.WaitForSafePP (0 ms) 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripPP 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1436: Skipped 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP (0 ms) 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripPP2 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1467: Skipped 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP2 (0 ms) 2024-08-16T02:41:32.942 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripPP3 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1500: Skipped 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP3 (0 ms) 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1605: Skipped 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripSparseReadPP (0 ms) 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1637: Skipped 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppendPP (0 ms) 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.IsCompletePP 2024-08-16T02:41:32.943 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1705: Skipped 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.IsCompletePP (0 ms) 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.IsSafePP 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1742: Skipped 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.IsSafePP (0 ms) 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.ReturnValuePP 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1780: Skipped 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValuePP (0 ms) 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.FlushPP 2024-08-16T02:41:32.944 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1796: Skipped 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.FlushPP (0 ms) 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.FlushAsyncPP 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1824: Skipped 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsyncPP (0 ms) 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1859: Skipped 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP (0 ms) 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1904: Skipped 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.945 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP2 (0 ms) 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.SimpleStatPP 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1946: Skipped 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPP (0 ms) 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1977: Skipped 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPPNS (0 ms) 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.StatRemovePP 2024-08-16T02:41:32.946 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2008: Skipped 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.StatRemovePP (0 ms) 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.ExecuteClassPP 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2059: Skipped 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClassPP (0 ms) 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.OmapPP 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2089: Skipped 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.947 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.OmapPP (0 ms) 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[ RUN ] LibRadosAioEC.MultiWritePP 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2124: Skipped 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.MultiWritePP (0 ms) 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] 20 tests from LibRadosAioEC (1 ms total) 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout: 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[----------] Global test environment tear-down 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[==========] 57 tests from 4 test suites ran. (161109 ms total) 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[ PASSED ] 35 tests. 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] 21 tests, listed below: 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAio.MultiReads 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWritePP 2024-08-16T02:41:32.948 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.WaitForSafePP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP2 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP3 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripSparseReadPP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppendPP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.IsCompletePP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.IsSafePP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValuePP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.FlushPP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsyncPP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP2 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPP 2024-08-16T02:41:32.949 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPPNS 2024-08-16T02:41:32.950 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.StatRemovePP 2024-08-16T02:41:32.950 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClassPP 2024-08-16T02:41:32.950 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.OmapPP 2024-08-16T02:41:32.951 INFO:tasks.workunit.client.0.smithi133.stdout:[ SKIPPED ] LibRadosAioEC.MultiWritePP 2024-08-16T02:41:32.951 INFO:tasks.workunit.client.0.smithi133.stdout:[ FAILED ] 1 test, listed below: 2024-08-16T02:41:32.951 INFO:tasks.workunit.client.0.smithi133.stdout:[ FAILED ] LibRadosAio.PoolEIOFlag 2024-08-16T02:41:32.951 INFO:tasks.workunit.client.0.smithi133.stdout: 2024-08-16T02:41:32.951 INFO:tasks.workunit.client.0.smithi133.stdout: 1 FAILED TEST 2024-08-16T02:41:32.951 INFO:tasks.workunit.client.0.smithi133.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl< >::~mutex_debug_impl() [with bool Recursive = false]' thread 7fc235e33800 time 2024-08-16T02:41:32.940385+0000 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/common/mutex_debug.h: 110: FAILED ceph_assert(r == 0) 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: ceph version 19.0.0-5812-g6f386a3f (6f386a3f0b1fe6a4d7219b54bf963d790bc84004) squid (dev) 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x129) [0x7fc23507906e] 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc2350792b4] 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: 3: ceph_test_rados_api_aio_pp() [0x4a3035] 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: 4: /lib64/libc.so.6(+0x40ded) [0x7fc233e40ded] 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: 5: on_exit() 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: 6: /lib64/libc.so.6(+0x29597) [0x7fc233e29597] 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: 7: __libc_start_main() 2024-08-16T02:41:32.952 INFO:tasks.workunit.client.0.smithi133.stderr: 8: _start() 2024-08-16T02:41:33.453 INFO:tasks.workunit.client.0.smithi133.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 24: 32181 Aborted (core dumped) ceph_test_rados_$f 2024-08-16T02:41:33.453 INFO:tasks.workunit.client.0.smithi133.stderr:++ cleanup 2024-08-16T02:41:33.454 INFO:tasks.workunit.client.0.smithi133.stderr:++ pkill -P 31708 2024-08-16T02:41:33.479 INFO:tasks.workunit.client.0.smithi133.stderr:++ true 2024-08-16T02:41:33.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ cleanup 2024-08-16T02:41:33.480 INFO:tasks.workunit.client.0.smithi133.stderr:+ pkill -P 31708 2024-08-16T02:41:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 134 2024-08-16T02:41:33.483 INFO:tasks.workunit.client.0.smithi133.stderr:+ true 2024-08-16T02:41:33.483 INFO:tasks.workunit:Stopping ['rados/test.sh --crimson', 'rados/test_pool_quota.sh'] on client.0... 2024-08-16T02:41:33.483 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-16T02:41:33.801 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi133 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 2024-08-16T02:41:33.993 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f8888b9dc1fa43dfb03b15daacc48ff5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi133 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 2024-08-16T02:41:33.995 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-16T02:41:34.004 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-16T02:41:34.005 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:41:34.118 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.114+0000 7f9bd0192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:34.119 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.118+0000 7f9bd0192640 1 Processor -- start 2024-08-16T02:41:34.119 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.119+0000 7f9bd0192640 1 -- start start 2024-08-16T02:41:34.121 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bd0192640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9bc81384a0 0x7f9bc813a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.121 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bd0192640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bc806cf10 con 0x7f9bc8135600 2024-08-16T02:41:34.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bd0192640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f9bc8136120 con 0x7f9bc81384a0 2024-08-16T02:41:34.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bcd72d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9bc81384a0 0x7f9bc813a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bcdf2e640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f9bc8135600 0x7f9bc8135a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32942/0 (socket says 172.21.15.133:32942) 2024-08-16T02:41:34.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bcd72d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9bc81384a0 0x7f9bc813a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33430/0 (socket says 172.21.15.133:33430) 2024-08-16T02:41:34.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bcdf2e640 1 -- 172.21.15.133:0/2471963506 learned_addr learned my addr 172.21.15.133:0/2471963506 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:34.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bcd72d640 1 -- 172.21.15.133:0/2471963506 >> v1:172.21.15.133:6789/0 conn(0x7f9bc8135600 legacy=0x7f9bc8135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bcd72d640 1 -- 172.21.15.133:0/2471963506 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc81351d0 con 0x7f9bc81384a0 2024-08-16T02:41:34.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bccf2c640 1 -- 172.21.15.133:0/2471963506 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3517430847 0 0) 0x7f9bc806cf10 con 0x7f9bc8135600 2024-08-16T02:41:34.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bcd72d640 1 --2- 172.21.15.133:0/2471963506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc81384a0 0x7f9bc813a8c0 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabb1e7dbab51dbf server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.120+0000 7f9bccf2c640 1 -- 172.21.15.133:0/2471963506 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9bb8009d90 con 0x7f9bc81384a0 2024-08-16T02:41:34.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bccf2c640 1 -- 172.21.15.133:0/2471963506 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f9bb800e230 con 0x7f9bc81384a0 2024-08-16T02:41:34.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bccf2c640 1 -- 172.21.15.133:0/2471963506 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9bb800e4d0 con 0x7f9bc81384a0 2024-08-16T02:41:34.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bd0192640 1 -- 172.21.15.133:0/2471963506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc81384a0 msgr2=0x7f9bc813a8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.126 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bd0192640 1 --2- 172.21.15.133:0/2471963506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc81384a0 0x7f9bc813a8c0 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.126 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bd0192640 1 -- 172.21.15.133:0/2471963506 shutdown_connections 2024-08-16T02:41:34.126 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bd0192640 1 --2- 172.21.15.133:0/2471963506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc81384a0 0x7f9bc813a8c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.126 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bd0192640 1 -- 172.21.15.133:0/2471963506 >> 172.21.15.133:0/2471963506 conn(0x7f9bc806ec10 msgr2=0x7f9bc806f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:34.126 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bd0192640 1 -- 172.21.15.133:0/2471963506 shutdown_connections 2024-08-16T02:41:34.126 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.121+0000 7f9bd0192640 1 -- 172.21.15.133:0/2471963506 wait complete. 2024-08-16T02:41:34.126 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.122+0000 7f9bd0192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:34.126 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.123+0000 7f9bd0192640 1 Processor -- start 2024-08-16T02:41:34.127 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.123+0000 7f9bd0192640 1 -- start start 2024-08-16T02:41:34.127 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.124+0000 7f9bd0192640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc8135600 0x7f9bc811fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.127 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.124+0000 7f9bd0192640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc8120d80 con 0x7f9bc8135600 2024-08-16T02:41:34.127 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.124+0000 7f9bcdf2e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc8135600 0x7f9bc811fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.127 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.124+0000 7f9bcdf2e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc8135600 0x7f9bc811fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33432/0 (socket says 172.21.15.133:33432) 2024-08-16T02:41:34.127 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.124+0000 7f9bcdf2e640 1 -- 172.21.15.133:0/2970125943 learned_addr learned my addr 172.21.15.133:0/2970125943 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:34.127 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.124+0000 7f9bcdf2e640 1 -- 172.21.15.133:0/2970125943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc8121ba0 con 0x7f9bc8135600 2024-08-16T02:41:34.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.124+0000 7f9bcdf2e640 1 --2- 172.21.15.133:0/2970125943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc8135600 0x7f9bc811fc80 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.124+0000 7f9bbe7fc640 1 -- 172.21.15.133:0/2970125943 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9bc4004b80 con 0x7f9bc8135600 2024-08-16T02:41:34.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.125+0000 7f9bbe7fc640 1 -- 172.21.15.133:0/2970125943 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f9bc4004ce0 con 0x7f9bc8135600 2024-08-16T02:41:34.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.125+0000 7f9bbe7fc640 1 -- 172.21.15.133:0/2970125943 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9bc4007120 con 0x7f9bc8135600 2024-08-16T02:41:34.130 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.125+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc811f340 con 0x7f9bc8135600 2024-08-16T02:41:34.130 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.125+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc8122700 con 0x7f9bc8135600 2024-08-16T02:41:34.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.125+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc806a020 con 0x7f9bc8135600 2024-08-16T02:41:34.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.126+0000 7f9bbe7fc640 1 -- 172.21.15.133:0/2970125943 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f9bc40072d0 con 0x7f9bc8135600 2024-08-16T02:41:34.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.126+0000 7f9bbe7fc640 1 --2- 172.21.15.133:0/2970125943 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f9bac03dd10 0x7f9bac040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.126+0000 7f9bbe7fc640 1 -- 172.21.15.133:0/2970125943 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f9bc404e790 con 0x7f9bc8135600 2024-08-16T02:41:34.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.126+0000 7f9bcd72d640 1 --2- 172.21.15.133:0/2970125943 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f9bac03dd10 0x7f9bac040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.127+0000 7f9bcd72d640 1 --2- 172.21.15.133:0/2970125943 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f9bac03dd10 0x7f9bac040220 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.131+0000 7f9bbe7fc640 1 -- 172.21.15.133:0/2970125943 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f9bc400adb0 con 0x7f9bc8135600 2024-08-16T02:41:34.314 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.313+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9bc809d420 con 0x7f9bac03dd10 2024-08-16T02:41:34.315 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.315+0000 7f9bbe7fc640 1 -- 172.21.15.133:0/2970125943 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7f9bb8009e80 con 0x7f9bac03dd10 2024-08-16T02:41:34.316 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:41:34.316 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-08-16T02:41:34.318 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f9bac03dd10 msgr2=0x7f9bac040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.318 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 --2- 172.21.15.133:0/2970125943 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f9bac03dd10 0x7f9bac040220 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.318 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc8135600 msgr2=0x7f9bc811fc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.318 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 --2- 172.21.15.133:0/2970125943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc8135600 0x7f9bc811fc80 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.319 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 shutdown_connections 2024-08-16T02:41:34.319 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 --2- 172.21.15.133:0/2970125943 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f9bac03dd10 0x7f9bac040220 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.319 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 --2- 172.21.15.133:0/2970125943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bc8135600 0x7f9bc811fc80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.319 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 >> 172.21.15.133:0/2970125943 conn(0x7f9bc806ec10 msgr2=0x7f9bc8133e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:34.319 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 shutdown_connections 2024-08-16T02:41:34.319 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.317+0000 7f9bd0192640 1 -- 172.21.15.133:0/2970125943 wait complete. 2024-08-16T02:41:34.329 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-08-16T02:41:33.441131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":16800,"kb_used_data":16800,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209174112,"statfs":{"total":214211493888,"available":214194290688,"internally_reserved":0,"allocated":17203200,"data_stored":17203200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000713"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.950223+0000","last_change":"2024-08-16T02:36:32.491674+0000","last_active":"2024-08-16T02:41:29.950223+0000","last_peered":"2024-08-16T02:41:29.950223+0000","last_clean":"2024-08-16T02:41:29.950223+0000","last_became_active":"2024-08-16T02:36:30.530297+0000","last_became_peered":"2024-08-16T02:36:30.530297+0000","last_unstale":"2024-08-16T02:41:29.950223+0000","last_undegraded":"2024-08-16T02:41:29.950223+0000","last_fullsized":"2024-08-16T02:41:29.950223+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.979304+0000","last_change":"2024-08-16T02:36:32.493152+0000","last_active":"2024-08-16T02:41:29.979304+0000","last_peered":"2024-08-16T02:41:29.979304+0000","last_clean":"2024-08-16T02:41:29.979304+0000","last_became_active":"2024-08-16T02:36:30.537306+0000","last_became_peered":"2024-08-16T02:36:30.537306+0000","last_unstale":"2024-08-16T02:41:29.979304+0000","last_undegraded":"2024-08-16T02:41:29.979304+0000","last_fullsized":"2024-08-16T02:41:29.979304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:30.001975+0000","last_change":"2024-08-16T02:36:32.491067+0000","last_active":"2024-08-16T02:41:30.001975+0000","last_peered":"2024-08-16T02:41:30.001975+0000","last_clean":"2024-08-16T02:41:30.001975+0000","last_became_active":"2024-08-16T02:36:30.537547+0000","last_became_peered":"2024-08-16T02:36:30.537547+0000","last_unstale":"2024-08-16T02:41:30.001975+0000","last_undegraded":"2024-08-16T02:41:30.001975+0000","last_fullsized":"2024-08-16T02:41:30.001975+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.953413+0000","last_change":"2024-08-16T02:36:32.489861+0000","last_active":"2024-08-16T02:41:29.953413+0000","last_peered":"2024-08-16T02:41:29.953413+0000","last_clean":"2024-08-16T02:41:29.953413+0000","last_became_active":"2024-08-16T02:36:30.521009+0000","last_became_peered":"2024-08-16T02:36:30.521009+0000","last_unstale":"2024-08-16T02:41:29.953413+0000","last_undegraded":"2024-08-16T02:41:29.953413+0000","last_fullsized":"2024-08-16T02:41:29.953413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.974040+0000","last_change":"2024-08-16T02:36:32.489707+0000","last_active":"2024-08-16T02:41:29.974040+0000","last_peered":"2024-08-16T02:41:29.974040+0000","last_clean":"2024-08-16T02:41:29.974040+0000","last_became_active":"2024-08-16T02:36:30.539438+0000","last_became_peered":"2024-08-16T02:36:30.539438+0000","last_unstale":"2024-08-16T02:41:29.974040+0000","last_undegraded":"2024-08-16T02:41:29.974040+0000","last_fullsized":"2024-08-16T02:41:29.974040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.962629+0000","last_change":"2024-08-16T02:36:32.491302+0000","last_active":"2024-08-16T02:41:29.962629+0000","last_peered":"2024-08-16T02:41:29.962629+0000","last_clean":"2024-08-16T02:41:29.962629+0000","last_became_active":"2024-08-16T02:36:30.521005+0000","last_became_peered":"2024-08-16T02:36:30.521005+0000","last_unstale":"2024-08-16T02:41:29.962629+0000","last_undegraded":"2024-08-16T02:41:29.962629+0000","last_fullsized":"2024-08-16T02:41:29.962629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.983863+0000","last_change":"2024-08-16T02:36:32.490191+0000","last_active":"2024-08-16T02:41:29.983863+0000","last_peered":"2024-08-16T02:41:29.983863+0000","last_clean":"2024-08-16T02:41:29.983863+0000","last_became_active":"2024-08-16T02:36:30.548723+0000","last_became_peered":"2024-08-16T02:36:30.548723+0000","last_unstale":"2024-08-16T02:41:29.983863+0000","last_undegraded":"2024-08-16T02:41:29.983863+0000","last_fullsized":"2024-08-16T02:41:29.983863+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:30.005020+0000","last_change":"2024-08-16T02:36:32.495617+0000","last_active":"2024-08-16T02:41:30.005020+0000","last_peered":"2024-08-16T02:41:30.005020+0000","last_clean":"2024-08-16T02:41:30.005020+0000","last_became_active":"2024-08-16T02:36:30.528787+0000","last_became_peered":"2024-08-16T02:36:30.528787+0000","last_unstale":"2024-08-16T02:41:30.005020+0000","last_undegraded":"2024-08-16T02:41:30.005020+0000","last_fullsized":"2024-08-16T02:41:30.005020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.965506+0000","last_change":"2024-08-16T02:36:28.507195+0000","last_active":"2024-08-16T02:41:29.965506+0000","last_peered":"2024-08-16T02:41:29.965506+0000","last_clean":"2024-08-16T02:41:29.965506+0000","last_became_active":"2024-08-16T02:36:28.489045+0000","last_became_peered":"2024-08-16T02:36:28.489045+0000","last_unstale":"2024-08-16T02:41:29.965506+0000","last_undegraded":"2024-08-16T02:41:29.965506+0000","last_fullsized":"2024-08-16T02:41:29.965506+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_clean_scrub_stamp":"2024-08-16T02:36:27.432441+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198435840,"internally_reserved":0,"allocated":13058048,"data_stored":13058048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4584,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293144,"statfs":{"total":53552873472,"available":53548179456,"internally_reserved":0,"allocated":4694016,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3672,"kb_used_data":3672,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294056,"statfs":{"total":53552873472,"available":53549113344,"internally_reserved":0,"allocated":3760128,"data_stored":3760128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3684,"kb_used_data":3684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294044,"statfs":{"total":53552873472,"available":53549101056,"internally_reserved":0,"allocated":3772416,"data_stored":3772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4860,"kb_used_data":4860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292868,"statfs":{"total":53552873472,"available":53547896832,"internally_reserved":0,"allocated":4976640,"data_stored":4976640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550018560,"internally_reserved":0,"allocated":2854912,"data_stored":2854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:41:34.330 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:41:34.439 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.434+0000 7f153e4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:34.440 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.440+0000 7f153e4ad640 1 Processor -- start 2024-08-16T02:41:34.441 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.440+0000 7f153e4ad640 1 -- start start 2024-08-16T02:41:34.442 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.441+0000 7f153e4ad640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1538133120 0x7f1538133530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.441+0000 7f153e4ad640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f153809bca0 con 0x7f1538133a70 2024-08-16T02:41:34.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.441+0000 7f153e4ad640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f15380a1fe0 con 0x7f1538133120 2024-08-16T02:41:34.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.441+0000 7f1537fff640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1538133120 0x7f1538133530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.441+0000 7f152ffff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f1538133a70 0x7f153812fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32948/0 (socket says 172.21.15.133:32948) 2024-08-16T02:41:34.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.441+0000 7f152ffff640 1 -- 172.21.15.133:0/4058171559 learned_addr learned my addr 172.21.15.133:0/4058171559 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:34.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.441+0000 7f1537fff640 1 -- 172.21.15.133:0/4058171559 >> v1:172.21.15.133:6789/0 conn(0x7f1538133a70 legacy=0x7f153812fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.441+0000 7f1537fff640 1 -- 172.21.15.133:0/4058171559 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15380746c0 con 0x7f1538133120 2024-08-16T02:41:34.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.442+0000 7f1537fff640 1 --2- 172.21.15.133:0/4058171559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f1538133530 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca169da17d3694f7 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.442+0000 7f15377fe640 1 -- 172.21.15.133:0/4058171559 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1528009d90 con 0x7f1538133120 2024-08-16T02:41:34.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.442+0000 7f15377fe640 1 -- 172.21.15.133:0/4058171559 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f152800e230 con 0x7f1538133120 2024-08-16T02:41:34.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.442+0000 7f15377fe640 1 -- 172.21.15.133:0/4058171559 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f152800e4d0 con 0x7f1538133120 2024-08-16T02:41:34.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.442+0000 7f153e4ad640 1 -- 172.21.15.133:0/4058171559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 msgr2=0x7f1538133530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.442+0000 7f153e4ad640 1 --2- 172.21.15.133:0/4058171559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f1538133530 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.443+0000 7f153e4ad640 1 -- 172.21.15.133:0/4058171559 shutdown_connections 2024-08-16T02:41:34.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.443+0000 7f153e4ad640 1 --2- 172.21.15.133:0/4058171559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f1538133530 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.443+0000 7f153e4ad640 1 -- 172.21.15.133:0/4058171559 >> 172.21.15.133:0/4058171559 conn(0x7f15380a21e0 msgr2=0x7f15380a25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:34.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.443+0000 7f153e4ad640 1 -- 172.21.15.133:0/4058171559 shutdown_connections 2024-08-16T02:41:34.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.443+0000 7f153e4ad640 1 -- 172.21.15.133:0/4058171559 wait complete. 2024-08-16T02:41:34.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.443+0000 7f153e4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:34.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.444+0000 7f153e4ad640 1 Processor -- start 2024-08-16T02:41:34.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.444+0000 7f153e4ad640 1 -- start start 2024-08-16T02:41:34.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.445+0000 7f153e4ad640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f153811bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.445+0000 7f153e4ad640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f153811f700 con 0x7f1538133120 2024-08-16T02:41:34.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.445+0000 7f1537fff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f153811bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.445+0000 7f1537fff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f153811bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33460/0 (socket says 172.21.15.133:33460) 2024-08-16T02:41:34.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.445+0000 7f1537fff640 1 -- 172.21.15.133:0/85596652 learned_addr learned my addr 172.21.15.133:0/85596652 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:34.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.445+0000 7f1537fff640 1 -- 172.21.15.133:0/85596652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153811d540 con 0x7f1538133120 2024-08-16T02:41:34.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.446+0000 7f1537fff640 1 --2- 172.21.15.133:0/85596652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f153811bd30 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.446+0000 7f15357fa640 1 -- 172.21.15.133:0/85596652 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1528014620 con 0x7f1538133120 2024-08-16T02:41:34.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.446+0000 7f15357fa640 1 -- 172.21.15.133:0/85596652 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1528014780 con 0x7f1538133120 2024-08-16T02:41:34.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.446+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153811de20 con 0x7f1538133120 2024-08-16T02:41:34.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.446+0000 7f15357fa640 1 -- 172.21.15.133:0/85596652 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1528014a20 con 0x7f1538133120 2024-08-16T02:41:34.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.446+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153812fd20 con 0x7f1538133120 2024-08-16T02:41:34.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.447+0000 7f15357fa640 1 -- 172.21.15.133:0/85596652 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f1528014bd0 con 0x7f1538133120 2024-08-16T02:41:34.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.447+0000 7f15357fa640 1 --2- 172.21.15.133:0/85596652 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f151803dd10 0x7f1518040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.447+0000 7f15357fa640 1 -- 172.21.15.133:0/85596652 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f1528019040 con 0x7f1538133120 2024-08-16T02:41:34.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.447+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1538077700 con 0x7f1538133120 2024-08-16T02:41:34.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.447+0000 7f152ffff640 1 --2- 172.21.15.133:0/85596652 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f151803dd10 0x7f1518040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.448+0000 7f152ffff640 1 --2- 172.21.15.133:0/85596652 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f151803dd10 0x7f1518040220 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.452+0000 7f15357fa640 1 -- 172.21.15.133:0/85596652 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f1528018970 con 0x7f1538133120 2024-08-16T02:41:34.651 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.650+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f153809cf90 con 0x7f151803dd10 2024-08-16T02:41:34.653 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.652+0000 7f15357fa640 1 -- 172.21.15.133:0/85596652 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7f1520005910 con 0x7f151803dd10 2024-08-16T02:41:34.653 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:41:34.653 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-08-16T02:41:34.655 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f151803dd10 msgr2=0x7f1518040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.655 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 --2- 172.21.15.133:0/85596652 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f151803dd10 0x7f1518040220 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 msgr2=0x7f153811bd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 --2- 172.21.15.133:0/85596652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f153811bd30 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 shutdown_connections 2024-08-16T02:41:34.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 --2- 172.21.15.133:0/85596652 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f151803dd10 0x7f1518040220 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 --2- 172.21.15.133:0/85596652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1538133120 0x7f153811bd30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 >> 172.21.15.133:0/85596652 conn(0x7f15380a21e0 msgr2=0x7f15380a35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:34.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 shutdown_connections 2024-08-16T02:41:34.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.654+0000 7f153e4ad640 1 -- 172.21.15.133:0/85596652 wait complete. 2024-08-16T02:41:34.666 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-08-16T02:41:33.441131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":16800,"kb_used_data":16800,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209174112,"statfs":{"total":214211493888,"available":214194290688,"internally_reserved":0,"allocated":17203200,"data_stored":17203200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000713"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.950223+0000","last_change":"2024-08-16T02:36:32.491674+0000","last_active":"2024-08-16T02:41:29.950223+0000","last_peered":"2024-08-16T02:41:29.950223+0000","last_clean":"2024-08-16T02:41:29.950223+0000","last_became_active":"2024-08-16T02:36:30.530297+0000","last_became_peered":"2024-08-16T02:36:30.530297+0000","last_unstale":"2024-08-16T02:41:29.950223+0000","last_undegraded":"2024-08-16T02:41:29.950223+0000","last_fullsized":"2024-08-16T02:41:29.950223+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.979304+0000","last_change":"2024-08-16T02:36:32.493152+0000","last_active":"2024-08-16T02:41:29.979304+0000","last_peered":"2024-08-16T02:41:29.979304+0000","last_clean":"2024-08-16T02:41:29.979304+0000","last_became_active":"2024-08-16T02:36:30.537306+0000","last_became_peered":"2024-08-16T02:36:30.537306+0000","last_unstale":"2024-08-16T02:41:29.979304+0000","last_undegraded":"2024-08-16T02:41:29.979304+0000","last_fullsized":"2024-08-16T02:41:29.979304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:30.001975+0000","last_change":"2024-08-16T02:36:32.491067+0000","last_active":"2024-08-16T02:41:30.001975+0000","last_peered":"2024-08-16T02:41:30.001975+0000","last_clean":"2024-08-16T02:41:30.001975+0000","last_became_active":"2024-08-16T02:36:30.537547+0000","last_became_peered":"2024-08-16T02:36:30.537547+0000","last_unstale":"2024-08-16T02:41:30.001975+0000","last_undegraded":"2024-08-16T02:41:30.001975+0000","last_fullsized":"2024-08-16T02:41:30.001975+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.953413+0000","last_change":"2024-08-16T02:36:32.489861+0000","last_active":"2024-08-16T02:41:29.953413+0000","last_peered":"2024-08-16T02:41:29.953413+0000","last_clean":"2024-08-16T02:41:29.953413+0000","last_became_active":"2024-08-16T02:36:30.521009+0000","last_became_peered":"2024-08-16T02:36:30.521009+0000","last_unstale":"2024-08-16T02:41:29.953413+0000","last_undegraded":"2024-08-16T02:41:29.953413+0000","last_fullsized":"2024-08-16T02:41:29.953413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.974040+0000","last_change":"2024-08-16T02:36:32.489707+0000","last_active":"2024-08-16T02:41:29.974040+0000","last_peered":"2024-08-16T02:41:29.974040+0000","last_clean":"2024-08-16T02:41:29.974040+0000","last_became_active":"2024-08-16T02:36:30.539438+0000","last_became_peered":"2024-08-16T02:36:30.539438+0000","last_unstale":"2024-08-16T02:41:29.974040+0000","last_undegraded":"2024-08-16T02:41:29.974040+0000","last_fullsized":"2024-08-16T02:41:29.974040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.962629+0000","last_change":"2024-08-16T02:36:32.491302+0000","last_active":"2024-08-16T02:41:29.962629+0000","last_peered":"2024-08-16T02:41:29.962629+0000","last_clean":"2024-08-16T02:41:29.962629+0000","last_became_active":"2024-08-16T02:36:30.521005+0000","last_became_peered":"2024-08-16T02:36:30.521005+0000","last_unstale":"2024-08-16T02:41:29.962629+0000","last_undegraded":"2024-08-16T02:41:29.962629+0000","last_fullsized":"2024-08-16T02:41:29.962629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.983863+0000","last_change":"2024-08-16T02:36:32.490191+0000","last_active":"2024-08-16T02:41:29.983863+0000","last_peered":"2024-08-16T02:41:29.983863+0000","last_clean":"2024-08-16T02:41:29.983863+0000","last_became_active":"2024-08-16T02:36:30.548723+0000","last_became_peered":"2024-08-16T02:36:30.548723+0000","last_unstale":"2024-08-16T02:41:29.983863+0000","last_undegraded":"2024-08-16T02:41:29.983863+0000","last_fullsized":"2024-08-16T02:41:29.983863+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:30.005020+0000","last_change":"2024-08-16T02:36:32.495617+0000","last_active":"2024-08-16T02:41:30.005020+0000","last_peered":"2024-08-16T02:41:30.005020+0000","last_clean":"2024-08-16T02:41:30.005020+0000","last_became_active":"2024-08-16T02:36:30.528787+0000","last_became_peered":"2024-08-16T02:36:30.528787+0000","last_unstale":"2024-08-16T02:41:30.005020+0000","last_undegraded":"2024-08-16T02:41:30.005020+0000","last_fullsized":"2024-08-16T02:41:30.005020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.965506+0000","last_change":"2024-08-16T02:36:28.507195+0000","last_active":"2024-08-16T02:41:29.965506+0000","last_peered":"2024-08-16T02:41:29.965506+0000","last_clean":"2024-08-16T02:41:29.965506+0000","last_became_active":"2024-08-16T02:36:28.489045+0000","last_became_peered":"2024-08-16T02:36:28.489045+0000","last_unstale":"2024-08-16T02:41:29.965506+0000","last_undegraded":"2024-08-16T02:41:29.965506+0000","last_fullsized":"2024-08-16T02:41:29.965506+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_clean_scrub_stamp":"2024-08-16T02:36:27.432441+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198435840,"internally_reserved":0,"allocated":13058048,"data_stored":13058048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4584,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293144,"statfs":{"total":53552873472,"available":53548179456,"internally_reserved":0,"allocated":4694016,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3672,"kb_used_data":3672,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294056,"statfs":{"total":53552873472,"available":53549113344,"internally_reserved":0,"allocated":3760128,"data_stored":3760128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3684,"kb_used_data":3684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294044,"statfs":{"total":53552873472,"available":53549101056,"internally_reserved":0,"allocated":3772416,"data_stored":3772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4860,"kb_used_data":4860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292868,"statfs":{"total":53552873472,"available":53547896832,"internally_reserved":0,"allocated":4976640,"data_stored":4976640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550018560,"internally_reserved":0,"allocated":2854912,"data_stored":2854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:41:34.667 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-16T02:41:34.667 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:41:34.776 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.772+0000 7fbe31792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:34.777 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.776+0000 7fbe31792640 1 Processor -- start 2024-08-16T02:41:34.777 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.776+0000 7fbe31792640 1 -- start start 2024-08-16T02:41:34.779 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe31792640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fbe2c09e340 0x7fbe2c138540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.780 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe31792640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe2c0a0f70 con 0x7fbe2c06f660 2024-08-16T02:41:34.780 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe31792640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7fbe2c0a0d00 con 0x7fbe2c09e340 2024-08-16T02:41:34.780 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe2a7fc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fbe2c09e340 0x7fbe2c138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.780 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe2affd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fbe2c06f660 0x7fbe2c09dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32952/0 (socket says 172.21.15.133:32952) 2024-08-16T02:41:34.780 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe2affd640 1 -- 172.21.15.133:0/462807447 learned_addr learned my addr 172.21.15.133:0/462807447 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:34.780 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe2a7fc640 1 -- 172.21.15.133:0/462807447 >> v1:172.21.15.133:6789/0 conn(0x7fbe2c06f660 legacy=0x7fbe2c09dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.780 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe2a7fc640 1 -- 172.21.15.133:0/462807447 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c09ee50 con 0x7fbe2c09e340 2024-08-16T02:41:34.780 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe29ffb640 1 -- 172.21.15.133:0/462807447 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054406165 0 0) 0x7fbe2c0a0f70 con 0x7fbe2c06f660 2024-08-16T02:41:34.781 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe2a7fc640 1 --2- 172.21.15.133:0/462807447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c09e340 0x7fbe2c138540 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5084ec234d39a0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.781 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe29ffb640 1 -- 172.21.15.133:0/462807447 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbe20009d90 con 0x7fbe2c09e340 2024-08-16T02:41:34.781 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe29ffb640 1 -- 172.21.15.133:0/462807447 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbe2000e230 con 0x7fbe2c09e340 2024-08-16T02:41:34.781 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.778+0000 7fbe29ffb640 1 -- 172.21.15.133:0/462807447 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbe2000e4d0 con 0x7fbe2c09e340 2024-08-16T02:41:34.782 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.779+0000 7fbe31792640 1 -- 172.21.15.133:0/462807447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c09e340 msgr2=0x7fbe2c138540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.782 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.779+0000 7fbe31792640 1 --2- 172.21.15.133:0/462807447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c09e340 0x7fbe2c138540 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.782 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.779+0000 7fbe31792640 1 -- 172.21.15.133:0/462807447 shutdown_connections 2024-08-16T02:41:34.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.779+0000 7fbe31792640 1 --2- 172.21.15.133:0/462807447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c09e340 0x7fbe2c138540 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.779+0000 7fbe31792640 1 -- 172.21.15.133:0/462807447 >> 172.21.15.133:0/462807447 conn(0x7fbe2c09f1d0 msgr2=0x7fbe2c09f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:34.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.779+0000 7fbe31792640 1 -- 172.21.15.133:0/462807447 shutdown_connections 2024-08-16T02:41:34.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.779+0000 7fbe31792640 1 -- 172.21.15.133:0/462807447 wait complete. 2024-08-16T02:41:34.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.779+0000 7fbe31792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:34.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.781+0000 7fbe31792640 1 Processor -- start 2024-08-16T02:41:34.783 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.781+0000 7fbe31792640 1 -- start start 2024-08-16T02:41:34.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.782+0000 7fbe31792640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c06f660 0x7fbe2c11fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.785 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.782+0000 7fbe31792640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c120e40 con 0x7fbe2c06f660 2024-08-16T02:41:34.786 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.782+0000 7fbe2affd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c06f660 0x7fbe2c11fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.786 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.782+0000 7fbe2affd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c06f660 0x7fbe2c11fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33470/0 (socket says 172.21.15.133:33470) 2024-08-16T02:41:34.786 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.782+0000 7fbe2affd640 1 -- 172.21.15.133:0/2570674918 learned_addr learned my addr 172.21.15.133:0/2570674918 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:34.786 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.783+0000 7fbe2affd640 1 -- 172.21.15.133:0/2570674918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c070f20 con 0x7fbe2c06f660 2024-08-16T02:41:34.786 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.783+0000 7fbe2affd640 1 --2- 172.21.15.133:0/2570674918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c06f660 0x7fbe2c11fb00 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.786 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.783+0000 7fbe137fe640 1 -- 172.21.15.133:0/2570674918 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbe1c004b80 con 0x7fbe2c06f660 2024-08-16T02:41:34.786 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.783+0000 7fbe137fe640 1 -- 172.21.15.133:0/2570674918 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbe1c004ce0 con 0x7fbe2c06f660 2024-08-16T02:41:34.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.783+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2c123ae0 con 0x7fbe2c06f660 2024-08-16T02:41:34.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.783+0000 7fbe137fe640 1 -- 172.21.15.133:0/2570674918 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbe1c007120 con 0x7fbe2c06f660 2024-08-16T02:41:34.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.783+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe2c133d80 con 0x7fbe2c06f660 2024-08-16T02:41:34.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.784+0000 7fbe137fe640 1 -- 172.21.15.133:0/2570674918 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fbe1c0072d0 con 0x7fbe2c06f660 2024-08-16T02:41:34.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.784+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2c121db0 con 0x7fbe2c06f660 2024-08-16T02:41:34.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.784+0000 7fbe137fe640 1 --2- 172.21.15.133:0/2570674918 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fbe1803dd40 0x7fbe18040250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:34.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.784+0000 7fbe137fe640 1 -- 172.21.15.133:0/2570674918 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7fbe1c04e920 con 0x7fbe2c06f660 2024-08-16T02:41:34.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.785+0000 7fbe2a7fc640 1 --2- 172.21.15.133:0/2570674918 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fbe1803dd40 0x7fbe18040250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:34.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.785+0000 7fbe2a7fc640 1 --2- 172.21.15.133:0/2570674918 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fbe1803dd40 0x7fbe18040250 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:34.790 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.789+0000 7fbe137fe640 1 -- 172.21.15.133:0/2570674918 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fbe1c00adb0 con 0x7fbe2c06f660 2024-08-16T02:41:34.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.974+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe2c09dc30 con 0x7fbe1803dd40 2024-08-16T02:41:34.976 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.975+0000 7fbe137fe640 1 -- 172.21.15.133:0/2570674918 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7fbe2c09dc30 con 0x7fbe1803dd40 2024-08-16T02:41:34.977 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:41:34.977 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-08-16T02:41:34.978 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fbe1803dd40 msgr2=0x7fbe18040250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 --2- 172.21.15.133:0/2570674918 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fbe1803dd40 0x7fbe18040250 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c06f660 msgr2=0x7fbe2c11fb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:34.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 --2- 172.21.15.133:0/2570674918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c06f660 0x7fbe2c11fb00 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 shutdown_connections 2024-08-16T02:41:34.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 --2- 172.21.15.133:0/2570674918 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fbe1803dd40 0x7fbe18040250 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 --2- 172.21.15.133:0/2570674918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe2c06f660 0x7fbe2c11fb00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:34.979 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 >> 172.21.15.133:0/2570674918 conn(0x7fbe2c09f1d0 msgr2=0x7fbe2c1bb240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:34.980 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 shutdown_connections 2024-08-16T02:41:34.980 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:34.977+0000 7fbe31792640 1 -- 172.21.15.133:0/2570674918 wait complete. 2024-08-16T02:41:34.990 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-08-16T02:41:33.441131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":16800,"kb_used_data":16800,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209174112,"statfs":{"total":214211493888,"available":214194290688,"internally_reserved":0,"allocated":17203200,"data_stored":17203200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000713"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.950223+0000","last_change":"2024-08-16T02:36:32.491674+0000","last_active":"2024-08-16T02:41:29.950223+0000","last_peered":"2024-08-16T02:41:29.950223+0000","last_clean":"2024-08-16T02:41:29.950223+0000","last_became_active":"2024-08-16T02:36:30.530297+0000","last_became_peered":"2024-08-16T02:36:30.530297+0000","last_unstale":"2024-08-16T02:41:29.950223+0000","last_undegraded":"2024-08-16T02:41:29.950223+0000","last_fullsized":"2024-08-16T02:41:29.950223+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.979304+0000","last_change":"2024-08-16T02:36:32.493152+0000","last_active":"2024-08-16T02:41:29.979304+0000","last_peered":"2024-08-16T02:41:29.979304+0000","last_clean":"2024-08-16T02:41:29.979304+0000","last_became_active":"2024-08-16T02:36:30.537306+0000","last_became_peered":"2024-08-16T02:36:30.537306+0000","last_unstale":"2024-08-16T02:41:29.979304+0000","last_undegraded":"2024-08-16T02:41:29.979304+0000","last_fullsized":"2024-08-16T02:41:29.979304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:30.001975+0000","last_change":"2024-08-16T02:36:32.491067+0000","last_active":"2024-08-16T02:41:30.001975+0000","last_peered":"2024-08-16T02:41:30.001975+0000","last_clean":"2024-08-16T02:41:30.001975+0000","last_became_active":"2024-08-16T02:36:30.537547+0000","last_became_peered":"2024-08-16T02:36:30.537547+0000","last_unstale":"2024-08-16T02:41:30.001975+0000","last_undegraded":"2024-08-16T02:41:30.001975+0000","last_fullsized":"2024-08-16T02:41:30.001975+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.953413+0000","last_change":"2024-08-16T02:36:32.489861+0000","last_active":"2024-08-16T02:41:29.953413+0000","last_peered":"2024-08-16T02:41:29.953413+0000","last_clean":"2024-08-16T02:41:29.953413+0000","last_became_active":"2024-08-16T02:36:30.521009+0000","last_became_peered":"2024-08-16T02:36:30.521009+0000","last_unstale":"2024-08-16T02:41:29.953413+0000","last_undegraded":"2024-08-16T02:41:29.953413+0000","last_fullsized":"2024-08-16T02:41:29.953413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.974040+0000","last_change":"2024-08-16T02:36:32.489707+0000","last_active":"2024-08-16T02:41:29.974040+0000","last_peered":"2024-08-16T02:41:29.974040+0000","last_clean":"2024-08-16T02:41:29.974040+0000","last_became_active":"2024-08-16T02:36:30.539438+0000","last_became_peered":"2024-08-16T02:36:30.539438+0000","last_unstale":"2024-08-16T02:41:29.974040+0000","last_undegraded":"2024-08-16T02:41:29.974040+0000","last_fullsized":"2024-08-16T02:41:29.974040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.962629+0000","last_change":"2024-08-16T02:36:32.491302+0000","last_active":"2024-08-16T02:41:29.962629+0000","last_peered":"2024-08-16T02:41:29.962629+0000","last_clean":"2024-08-16T02:41:29.962629+0000","last_became_active":"2024-08-16T02:36:30.521005+0000","last_became_peered":"2024-08-16T02:36:30.521005+0000","last_unstale":"2024-08-16T02:41:29.962629+0000","last_undegraded":"2024-08-16T02:41:29.962629+0000","last_fullsized":"2024-08-16T02:41:29.962629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.983863+0000","last_change":"2024-08-16T02:36:32.490191+0000","last_active":"2024-08-16T02:41:29.983863+0000","last_peered":"2024-08-16T02:41:29.983863+0000","last_clean":"2024-08-16T02:41:29.983863+0000","last_became_active":"2024-08-16T02:36:30.548723+0000","last_became_peered":"2024-08-16T02:36:30.548723+0000","last_unstale":"2024-08-16T02:41:29.983863+0000","last_undegraded":"2024-08-16T02:41:29.983863+0000","last_fullsized":"2024-08-16T02:41:29.983863+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:30.005020+0000","last_change":"2024-08-16T02:36:32.495617+0000","last_active":"2024-08-16T02:41:30.005020+0000","last_peered":"2024-08-16T02:41:30.005020+0000","last_clean":"2024-08-16T02:41:30.005020+0000","last_became_active":"2024-08-16T02:36:30.528787+0000","last_became_peered":"2024-08-16T02:36:30.528787+0000","last_unstale":"2024-08-16T02:41:30.005020+0000","last_undegraded":"2024-08-16T02:41:30.005020+0000","last_fullsized":"2024-08-16T02:41:30.005020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:41:29.965506+0000","last_change":"2024-08-16T02:36:28.507195+0000","last_active":"2024-08-16T02:41:29.965506+0000","last_peered":"2024-08-16T02:41:29.965506+0000","last_clean":"2024-08-16T02:41:29.965506+0000","last_became_active":"2024-08-16T02:36:28.489045+0000","last_became_peered":"2024-08-16T02:36:28.489045+0000","last_unstale":"2024-08-16T02:41:29.965506+0000","last_undegraded":"2024-08-16T02:41:29.965506+0000","last_fullsized":"2024-08-16T02:41:29.965506+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_clean_scrub_stamp":"2024-08-16T02:36:27.432441+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198435840,"internally_reserved":0,"allocated":13058048,"data_stored":13058048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4584,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293144,"statfs":{"total":53552873472,"available":53548179456,"internally_reserved":0,"allocated":4694016,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3672,"kb_used_data":3672,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294056,"statfs":{"total":53552873472,"available":53549113344,"internally_reserved":0,"allocated":3760128,"data_stored":3760128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3684,"kb_used_data":3684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294044,"statfs":{"total":53552873472,"available":53549101056,"internally_reserved":0,"allocated":3772416,"data_stored":3772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4860,"kb_used_data":4860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292868,"statfs":{"total":53552873472,"available":53547896832,"internally_reserved":0,"allocated":4976640,"data_stored":4976640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550018560,"internally_reserved":0,"allocated":2854912,"data_stored":2854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:41:34.991 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:41:35.100 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.096+0000 7f2305417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:35.101 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.101+0000 7f2305417640 1 Processor -- start 2024-08-16T02:41:35.102 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.101+0000 7f2305417640 1 -- start start 2024-08-16T02:41:35.103 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f2305417640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f23001333d0 0x7f23001337e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:35.104 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f2305417640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2300133010 con 0x7f2300071bd0 2024-08-16T02:41:35.104 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f2305417640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f23000a4db0 con 0x7f23001333d0 2024-08-16T02:41:35.104 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f22feffd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f23001333d0 0x7f23001337e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:35.104 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f22f7fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2300071bd0 0x7f2300071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32964/0 (socket says 172.21.15.133:32964) 2024-08-16T02:41:35.104 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f22f7fff640 1 -- 172.21.15.133:0/695449977 learned_addr learned my addr 172.21.15.133:0/695449977 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:35.104 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f22feffd640 1 -- 172.21.15.133:0/695449977 >> v1:172.21.15.133:6789/0 conn(0x7f2300071bd0 legacy=0x7f2300071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:35.104 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f22feffd640 1 -- 172.21.15.133:0/695449977 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2300073df0 con 0x7f23001333d0 2024-08-16T02:41:35.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.102+0000 7f22fe7fc640 1 -- 172.21.15.133:0/695449977 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4197988075 0 0) 0x7f2300133010 con 0x7f2300071bd0 2024-08-16T02:41:35.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.103+0000 7f22feffd640 1 --2- 172.21.15.133:0/695449977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f23001333d0 0x7f23001337e0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34041952a2254200 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:35.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.103+0000 7f22fe7fc640 1 -- 172.21.15.133:0/695449977 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f22f0009d90 con 0x7f23001333d0 2024-08-16T02:41:35.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.103+0000 7f22fe7fc640 1 -- 172.21.15.133:0/695449977 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f22f000e230 con 0x7f23001333d0 2024-08-16T02:41:35.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.103+0000 7f22fe7fc640 1 -- 172.21.15.133:0/695449977 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f22f000e4d0 con 0x7f23001333d0 2024-08-16T02:41:35.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.103+0000 7f2305417640 1 -- 172.21.15.133:0/695449977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f23001333d0 msgr2=0x7f23001337e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:35.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.103+0000 7f2305417640 1 --2- 172.21.15.133:0/695449977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f23001333d0 0x7f23001337e0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:35.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.103+0000 7f2305417640 1 -- 172.21.15.133:0/695449977 shutdown_connections 2024-08-16T02:41:35.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.104+0000 7f2305417640 1 --2- 172.21.15.133:0/695449977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f23001333d0 0x7f23001337e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:35.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.104+0000 7f2305417640 1 -- 172.21.15.133:0/695449977 >> 172.21.15.133:0/695449977 conn(0x7f23000a2270 msgr2=0x7f23000a2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:35.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.104+0000 7f2305417640 1 -- 172.21.15.133:0/695449977 shutdown_connections 2024-08-16T02:41:35.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.104+0000 7f2305417640 1 -- 172.21.15.133:0/695449977 wait complete. 2024-08-16T02:41:35.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.104+0000 7f2305417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:35.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.105+0000 7f2305417640 1 Processor -- start 2024-08-16T02:41:35.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.105+0000 7f2305417640 1 -- start start 2024-08-16T02:41:35.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.106+0000 7f2305417640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2300071bd0 0x7f2300131ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:35.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.106+0000 7f2305417640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f2300121b10 con 0x7f2300071bd0 2024-08-16T02:41:35.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.106+0000 7f22feffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2300071bd0 0x7f2300131ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:35.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.106+0000 7f22feffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2300071bd0 0x7f2300131ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33478/0 (socket says 172.21.15.133:33478) 2024-08-16T02:41:35.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.106+0000 7f22feffd640 1 -- 172.21.15.133:0/2450634363 learned_addr learned my addr 172.21.15.133:0/2450634363 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:35.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.107+0000 7f22feffd640 1 -- 172.21.15.133:0/2450634363 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230011ef00 con 0x7f2300071bd0 2024-08-16T02:41:35.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.107+0000 7f22feffd640 1 --2- 172.21.15.133:0/2450634363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2300071bd0 0x7f2300131ae0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:35.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.107+0000 7f22f77fe640 1 -- 172.21.15.133:0/2450634363 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f22f0013610 con 0x7f2300071bd0 2024-08-16T02:41:35.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.107+0000 7f22f77fe640 1 -- 172.21.15.133:0/2450634363 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f22f0013770 con 0x7f2300071bd0 2024-08-16T02:41:35.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.107+0000 7f22f77fe640 1 -- 172.21.15.133:0/2450634363 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f22f0013a10 con 0x7f2300071bd0 2024-08-16T02:41:35.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.107+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230011fdb0 con 0x7f2300071bd0 2024-08-16T02:41:35.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.107+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f230011f540 con 0x7f2300071bd0 2024-08-16T02:41:35.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.108+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f230006a020 con 0x7f2300071bd0 2024-08-16T02:41:35.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.108+0000 7f22f77fe640 1 -- 172.21.15.133:0/2450634363 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f22f0013c10 con 0x7f2300071bd0 2024-08-16T02:41:35.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.108+0000 7f22f77fe640 1 --2- 172.21.15.133:0/2450634363 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f22e003dcc0 0x7f22e00401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:35.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.108+0000 7f22f77fe640 1 -- 172.21.15.133:0/2450634363 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f22f0018040 con 0x7f2300071bd0 2024-08-16T02:41:35.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.109+0000 7f22f7fff640 1 --2- 172.21.15.133:0/2450634363 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f22e003dcc0 0x7f22e00401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:35.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.109+0000 7f22f7fff640 1 --2- 172.21.15.133:0/2450634363 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f22e003dcc0 0x7f22e00401d0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:35.114 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.113+0000 7f22f77fe640 1 -- 172.21.15.133:0/2450634363 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f22f0017190 con 0x7f2300071bd0 2024-08-16T02:41:35.299 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.298+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23000741a0 con 0x7f2300071bd0 2024-08-16T02:41:35.299 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.299+0000 7f22f77fe640 1 -- 172.21.15.133:0/2450634363 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) ==== 74+0+8846 (crc 0 0 0) 0x7f22f000f2a0 con 0x7f2300071bd0 2024-08-16T02:41:35.300 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:41:35.300 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":228,"fsid":"7a5c695b-6aa6-4555-8e56-4a247af82439","created":"2024-08-16T02:36:18.775904+0000","modified":"2024-08-16T02:41:32.924840+0000","last_up_change":"2024-08-16T02:36:25.430618+0000","last_in_change":"2024-08-16T02:36:20.935538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":72,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:36:27.400474+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:36:29.162647+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3108e258-24a8-4a0f-86a5-a3ee5db0a828","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":2249864834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6805","nonce":2249864834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":2249864834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6807","nonce":2249864834}]},"public_addr":"172.21.15.133:6804/2249864834","cluster_addr":"172.21.15.133:6805/2249864834","heartbeat_back_addr":"172.21.15.133:6802/2249864834","heartbeat_front_addr":"172.21.15.133:6807/2249864834","state":["exists","up"]},{"osd":1,"uuid":"103b6a98-7079-4079-a37b-4216a77b9205","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6803","nonce":2214080839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":2214080839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6811","nonce":2214080839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":2214080839}]},"public_addr":"172.21.15.133:6803/2214080839","cluster_addr":"172.21.15.133:6806/2214080839","heartbeat_back_addr":"172.21.15.133:6811/2214080839","heartbeat_front_addr":"172.21.15.133:6810/2214080839","state":["exists","up"]},{"osd":2,"uuid":"14d9be38-a2e8-482d-96b8-48359bed89bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":3166527287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6807","nonce":3166527287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6809","nonce":3166527287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3166527287}]},"public_addr":"172.21.15.137:6808/3166527287","cluster_addr":"172.21.15.137:6807/3166527287","heartbeat_back_addr":"172.21.15.137:6809/3166527287","heartbeat_front_addr":"172.21.15.137:6804/3166527287","state":["exists","up"]},{"osd":3,"uuid":"d03f5fd2-ce08-4ea9-8390-2efa9b037a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6805","nonce":34553420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":34553420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":34553420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":34553420}]},"public_addr":"172.21.15.137:6805/34553420","cluster_addr":"172.21.15.137:6806/34553420","heartbeat_back_addr":"172.21.15.137:6802/34553420","heartbeat_front_addr":"172.21.15.137:6803/34553420","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:41:35.302 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f22e003dcc0 msgr2=0x7f22e00401d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:35.302 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 --2- 172.21.15.133:0/2450634363 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f22e003dcc0 0x7f22e00401d0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:35.302 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2300071bd0 msgr2=0x7f2300131ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:35.302 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 --2- 172.21.15.133:0/2450634363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2300071bd0 0x7f2300131ae0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:35.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 shutdown_connections 2024-08-16T02:41:35.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 --2- 172.21.15.133:0/2450634363 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f22e003dcc0 0x7f22e00401d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:35.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 --2- 172.21.15.133:0/2450634363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2300071bd0 0x7f2300131ae0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:35.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 >> 172.21.15.133:0/2450634363 conn(0x7f23000a2270 msgr2=0x7f23000a00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:35.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 shutdown_connections 2024-08-16T02:41:35.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:35.301+0000 7f2305417640 1 -- 172.21.15.133:0/2450634363 wait complete. 2024-08-16T02:41:36.315 INFO:tasks.ceph:Scrubbing osd.0 2024-08-16T02:41:36.315 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:41:36.441 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.432+0000 7f1cabc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:36.442 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.441+0000 7f1cabc6d640 1 Processor -- start 2024-08-16T02:41:36.442 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.441+0000 7f1cabc6d640 1 -- start start 2024-08-16T02:41:36.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.442+0000 7f1cabc6d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f1ca40740d0 0x7f1ca4138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.442+0000 7f1cabc6d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ca406e8b0 con 0x7f1ca40735b0 2024-08-16T02:41:36.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.442+0000 7f1cabc6d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca406eb50 con 0x7f1ca40740d0 2024-08-16T02:41:36.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca9a09640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f1ca40735b0 0x7f1ca40739c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32972/0 (socket says 172.21.15.133:32972) 2024-08-16T02:41:36.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca9a09640 1 -- 172.21.15.133:0/306776971 learned_addr learned my addr 172.21.15.133:0/306776971 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:36.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca9208640 1 --2- 172.21.15.133:0/306776971 >> v2:172.21.15.133:3300/0 conn(0x7f1ca40740d0 0x7f1ca4138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca8a07640 1 -- 172.21.15.133:0/306776971 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3345283800 0 0) 0x7f1ca406e8b0 con 0x7f1ca40735b0 2024-08-16T02:41:36.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca8a07640 1 -- 172.21.15.133:0/306776971 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c90003580 con 0x7f1ca40735b0 2024-08-16T02:41:36.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca9208640 1 -- 172.21.15.133:0/306776971 >> v1:172.21.15.133:6789/0 conn(0x7f1ca40735b0 legacy=0x7f1ca40739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca9208640 1 -- 172.21.15.133:0/306776971 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca409c800 con 0x7f1ca40740d0 2024-08-16T02:41:36.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca9208640 1 --2- 172.21.15.133:0/306776971 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40740d0 0x7f1ca4138640 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1265098f054ac31 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca8a07640 1 -- 172.21.15.133:0/306776971 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1c94009d90 con 0x7f1ca40740d0 2024-08-16T02:41:36.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca8a07640 1 -- 172.21.15.133:0/306776971 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1c9400e230 con 0x7f1ca40740d0 2024-08-16T02:41:36.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.443+0000 7f1ca8a07640 1 -- 172.21.15.133:0/306776971 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1c9400e4d0 con 0x7f1ca40740d0 2024-08-16T02:41:36.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.444+0000 7f1cabc6d640 1 -- 172.21.15.133:0/306776971 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40740d0 msgr2=0x7f1ca4138640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.444+0000 7f1cabc6d640 1 --2- 172.21.15.133:0/306776971 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40740d0 0x7f1ca4138640 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.444+0000 7f1cabc6d640 1 -- 172.21.15.133:0/306776971 shutdown_connections 2024-08-16T02:41:36.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.444+0000 7f1cabc6d640 1 --2- 172.21.15.133:0/306776971 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40740d0 0x7f1ca4138640 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.444+0000 7f1cabc6d640 1 -- 172.21.15.133:0/306776971 >> 172.21.15.133:0/306776971 conn(0x7f1ca406caf0 msgr2=0x7f1ca406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:36.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.444+0000 7f1cabc6d640 1 -- 172.21.15.133:0/306776971 shutdown_connections 2024-08-16T02:41:36.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.444+0000 7f1cabc6d640 1 -- 172.21.15.133:0/306776971 wait complete. 2024-08-16T02:41:36.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.444+0000 7f1cabc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:36.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.446+0000 7f1cabc6d640 1 Processor -- start 2024-08-16T02:41:36.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.446+0000 7f1cabc6d640 1 -- start start 2024-08-16T02:41:36.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1cabc6d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40735b0 0x7f1ca41c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1cabc6d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca411e2c0 con 0x7f1ca40735b0 2024-08-16T02:41:36.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1ca9a09640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40735b0 0x7f1ca41c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1ca9a09640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40735b0 0x7f1ca41c7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33492/0 (socket says 172.21.15.133:33492) 2024-08-16T02:41:36.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1ca9a09640 1 -- 172.21.15.133:0/2558946202 learned_addr learned my addr 172.21.15.133:0/2558946202 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:36.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1ca9a09640 1 -- 172.21.15.133:0/2558946202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca4120820 con 0x7f1ca40735b0 2024-08-16T02:41:36.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1ca9a09640 1 --2- 172.21.15.133:0/2558946202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40735b0 0x7f1ca41c7c90 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1ca0005a60 con 0x7f1ca40735b0 2024-08-16T02:41:36.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1ca0005bc0 con 0x7f1ca40735b0 2024-08-16T02:41:36.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca411eb00 con 0x7f1ca40735b0 2024-08-16T02:41:36.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1ca0005e60 con 0x7f1ca40735b0 2024-08-16T02:41:36.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.447+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca411c8c0 con 0x7f1ca40735b0 2024-08-16T02:41:36.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.448+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1ca4138b80 con 0x7f1ca40735b0 2024-08-16T02:41:36.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.449+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f1ca00070b0 con 0x7f1ca40735b0 2024-08-16T02:41:36.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.449+0000 7f1c9a7fc640 1 --2- 172.21.15.133:0/2558946202 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f1c8803dd10 0x7f1c88040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.449+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f1ca004e6d0 con 0x7f1ca40735b0 2024-08-16T02:41:36.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.449+0000 7f1c9a7fc640 1 --2- 172.21.15.133:0/2558946202 >> v2:172.21.15.133:6804/2249864834 conn(0x7f1c88042d10 0x7f1c880451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.449+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 --> v2:172.21.15.133:6804/2249864834 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c88045850 con 0x7f1c88042d10 2024-08-16T02:41:36.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.449+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7f1ca0050bc0 con 0x7f1ca40735b0 2024-08-16T02:41:36.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.449+0000 7f1ca9208640 1 --2- 172.21.15.133:0/2558946202 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f1c8803dd10 0x7f1c88040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.450+0000 7f1ca9208640 1 --2- 172.21.15.133:0/2558946202 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f1c8803dd10 0x7f1c88040220 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.450+0000 7f1caa20a640 1 --2- 172.21.15.133:0/2558946202 >> v2:172.21.15.133:6804/2249864834 conn(0x7f1c88042d10 0x7f1c880451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:41:36.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.453+0000 7f1caa20a640 1 --2- 172.21.15.133:0/2558946202 >> v2:172.21.15.133:6804/2249864834 conn(0x7f1c88042d10 0x7f1c880451c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.455+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 <== osd.0 v2:172.21.15.133:6804/2249864834 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f1c88045850 con 0x7f1c88042d10 2024-08-16T02:41:36.459 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.459+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 --> v2:172.21.15.133:6804/2249864834 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1ca4131e10 con 0x7f1c88042d10 2024-08-16T02:41:36.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.460+0000 7f1c9a7fc640 1 -- 172.21.15.133:0/2558946202 <== osd.0 v2:172.21.15.133:6804/2249864834 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f1ca4131e10 con 0x7f1c88042d10 2024-08-16T02:41:36.461 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2024-08-16T02:41:36.462 INFO:teuthology.orchestra.run.smithi133.stdout: "success": "" 2024-08-16T02:41:36.462 INFO:teuthology.orchestra.run.smithi133.stdout:} 2024-08-16T02:41:36.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 >> v2:172.21.15.133:6804/2249864834 conn(0x7f1c88042d10 msgr2=0x7f1c880451c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 --2- 172.21.15.133:0/2558946202 >> v2:172.21.15.133:6804/2249864834 conn(0x7f1c88042d10 0x7f1c880451c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f1c8803dd10 msgr2=0x7f1c88040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 --2- 172.21.15.133:0/2558946202 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f1c8803dd10 0x7f1c88040220 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40735b0 msgr2=0x7f1ca41c7c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 --2- 172.21.15.133:0/2558946202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40735b0 0x7f1ca41c7c90 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 shutdown_connections 2024-08-16T02:41:36.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 --2- 172.21.15.133:0/2558946202 >> v2:172.21.15.133:6804/2249864834 conn(0x7f1c88042d10 0x7f1c880451c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 --2- 172.21.15.133:0/2558946202 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f1c8803dd10 0x7f1c88040220 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 --2- 172.21.15.133:0/2558946202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ca40735b0 0x7f1ca41c7c90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 >> 172.21.15.133:0/2558946202 conn(0x7f1ca406caf0 msgr2=0x7f1ca409f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:36.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 shutdown_connections 2024-08-16T02:41:36.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.461+0000 7f1cabc6d640 1 -- 172.21.15.133:0/2558946202 wait complete. 2024-08-16T02:41:36.475 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-16T02:41:36.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.580+0000 7f33ccb92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:36.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.585+0000 7f33ccb92640 1 Processor -- start 2024-08-16T02:41:36.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.585+0000 7f33ccb92640 1 -- start start 2024-08-16T02:41:36.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33ccb92640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f33c81321a0 0x7f33c81325b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33ccb92640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c809dc90 con 0x7f33c8132af0 2024-08-16T02:41:36.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33ccb92640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f33c809eab0 con 0x7f33c81321a0 2024-08-16T02:41:36.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c659c640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f33c81321a0 0x7f33c81325b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c5d9b640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f33c8132af0 0x7f33c812eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32982/0 (socket says 172.21.15.133:32982) 2024-08-16T02:41:36.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c5d9b640 1 -- 172.21.15.133:0/1367121847 learned_addr learned my addr 172.21.15.133:0/1367121847 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:36.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c659c640 1 -- 172.21.15.133:0/1367121847 >> v1:172.21.15.133:6789/0 conn(0x7f33c8132af0 legacy=0x7f33c812eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c659c640 1 -- 172.21.15.133:0/1367121847 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c806d990 con 0x7f33c81321a0 2024-08-16T02:41:36.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c559a640 1 -- 172.21.15.133:0/1367121847 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3411999050 0 0) 0x7f33c809dc90 con 0x7f33c8132af0 2024-08-16T02:41:36.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c659c640 1 --2- 172.21.15.133:0/1367121847 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81325b0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49454bbda401ce3 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c559a640 1 -- 172.21.15.133:0/1367121847 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f33b800e040 con 0x7f33c81321a0 2024-08-16T02:41:36.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c559a640 1 -- 172.21.15.133:0/1367121847 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f33b8007670 con 0x7f33c81321a0 2024-08-16T02:41:36.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.587+0000 7f33c559a640 1 -- 172.21.15.133:0/1367121847 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f33b8007910 con 0x7f33c81321a0 2024-08-16T02:41:36.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.588+0000 7f33ccb92640 1 -- 172.21.15.133:0/1367121847 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 msgr2=0x7f33c81325b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.588+0000 7f33ccb92640 1 --2- 172.21.15.133:0/1367121847 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81325b0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.588+0000 7f33ccb92640 1 -- 172.21.15.133:0/1367121847 shutdown_connections 2024-08-16T02:41:36.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.588+0000 7f33ccb92640 1 --2- 172.21.15.133:0/1367121847 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81325b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.588+0000 7f33ccb92640 1 -- 172.21.15.133:0/1367121847 >> 172.21.15.133:0/1367121847 conn(0x7f33c809f1d0 msgr2=0x7f33c809f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:36.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.588+0000 7f33ccb92640 1 -- 172.21.15.133:0/1367121847 shutdown_connections 2024-08-16T02:41:36.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.588+0000 7f33ccb92640 1 -- 172.21.15.133:0/1367121847 wait complete. 2024-08-16T02:41:36.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.588+0000 7f33ccb92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:36.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.590+0000 7f33ccb92640 1 Processor -- start 2024-08-16T02:41:36.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.590+0000 7f33ccb92640 1 -- start start 2024-08-16T02:41:36.594 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.591+0000 7f33ccb92640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.594 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.591+0000 7f33ccb92640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f33c811ddf0 con 0x7f33c81321a0 2024-08-16T02:41:36.594 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.591+0000 7f33c659c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.594 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.591+0000 7f33c659c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33508/0 (socket says 172.21.15.133:33508) 2024-08-16T02:41:36.594 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.591+0000 7f33c659c640 1 -- 172.21.15.133:0/2418604959 learned_addr learned my addr 172.21.15.133:0/2418604959 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:36.594 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.591+0000 7f33c659c640 1 -- 172.21.15.133:0/2418604959 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c809dec0 con 0x7f33c81321a0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.591+0000 7f33c659c640 1 --2- 172.21.15.133:0/2418604959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81c6c60 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.592+0000 7f33beffd640 1 -- 172.21.15.133:0/2418604959 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f33b800e040 con 0x7f33c81321a0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.592+0000 7f33beffd640 1 -- 172.21.15.133:0/2418604959 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f33b8012a90 con 0x7f33c81321a0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.592+0000 7f33beffd640 1 -- 172.21.15.133:0/2418604959 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f33b8012d30 con 0x7f33c81321a0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.592+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c811bd30 con 0x7f33c81321a0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.592+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c80a3aa0 con 0x7f33c81321a0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.593+0000 7f33beffd640 1 -- 172.21.15.133:0/2418604959 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f33b80153f0 con 0x7f33c81321a0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.593+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33c81325b0 con 0x7f33c81321a0 2024-08-16T02:41:36.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.593+0000 7f33beffd640 1 --2- 172.21.15.133:0/2418604959 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f33a803dd10 0x7f33a8040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.593+0000 7f33beffd640 1 -- 172.21.15.133:0/2418604959 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f33b804ffa0 con 0x7f33c81321a0 2024-08-16T02:41:36.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.593+0000 7f33c5d9b640 1 --2- 172.21.15.133:0/2418604959 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f33a803dd10 0x7f33a8040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.594+0000 7f33c5d9b640 1 --2- 172.21.15.133:0/2418604959 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f33a803dd10 0x7f33a8040220 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.599 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.598+0000 7f33beffd640 1 -- 172.21.15.133:0/2418604959 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f33b800e380 con 0x7f33c81321a0 2024-08-16T02:41:36.787 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.786+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f33c809e6f0 con 0x7f33a803dd10 2024-08-16T02:41:36.788 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.787+0000 7f33beffd640 1 -- 172.21.15.133:0/2418604959 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f33b0005930 con 0x7f33a803dd10 2024-08-16T02:41:36.788 INFO:teuthology.orchestra.run.smithi133.stderr:instructed osd(s) 0 to deep-scrub 2024-08-16T02:41:36.792 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f33a803dd10 msgr2=0x7f33a8040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.793 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 --2- 172.21.15.133:0/2418604959 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f33a803dd10 0x7f33a8040220 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.793 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 msgr2=0x7f33c81c6c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.793 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 --2- 172.21.15.133:0/2418604959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81c6c60 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.793 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 shutdown_connections 2024-08-16T02:41:36.793 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 --2- 172.21.15.133:0/2418604959 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f33a803dd10 0x7f33a8040220 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.793 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 --2- 172.21.15.133:0/2418604959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33c81321a0 0x7f33c81c6c60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.793 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 >> 172.21.15.133:0/2418604959 conn(0x7f33c809f1d0 msgr2=0x7f33c809c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:36.793 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 shutdown_connections 2024-08-16T02:41:36.794 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.791+0000 7f33ccb92640 1 -- 172.21.15.133:0/2418604959 wait complete. 2024-08-16T02:41:36.807 INFO:tasks.ceph:Scrubbing osd.1 2024-08-16T02:41:36.808 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:41:36.921 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.911+0000 7f488706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:36.922 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.921+0000 7f488706d640 1 Processor -- start 2024-08-16T02:41:36.922 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.921+0000 7f488706d640 1 -- start start 2024-08-16T02:41:36.924 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.922+0000 7f488706d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4880135800 0x7f4880135c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.924 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.922+0000 7f488706d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4880072d00 con 0x7f488009fe30 2024-08-16T02:41:36.924 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.922+0000 7f488706d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f4880072060 con 0x7f4880135800 2024-08-16T02:41:36.924 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.922+0000 7f4884e09640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4880135800 0x7f4880135c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.924 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.922+0000 7f487ffff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f488009fe30 0x7f48800a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32986/0 (socket says 172.21.15.133:32986) 2024-08-16T02:41:36.925 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.922+0000 7f487ffff640 1 -- 172.21.15.133:0/3491515010 learned_addr learned my addr 172.21.15.133:0/3491515010 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:36.925 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.923+0000 7f4884e09640 1 -- 172.21.15.133:0/3491515010 >> v1:172.21.15.133:6789/0 conn(0x7f488009fe30 legacy=0x7f48800a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.925 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.923+0000 7f4884e09640 1 -- 172.21.15.133:0/3491515010 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4880135320 con 0x7f4880135800 2024-08-16T02:41:36.925 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.923+0000 7f487f7fe640 1 -- 172.21.15.133:0/3491515010 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 21518486 0 0) 0x7f4880072d00 con 0x7f488009fe30 2024-08-16T02:41:36.925 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.923+0000 7f4884e09640 1 --2- 172.21.15.133:0/3491515010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4880135800 0x7f4880135c10 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e6f65de6d70f47 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.925 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.923+0000 7f487f7fe640 1 -- 172.21.15.133:0/3491515010 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4878009d90 con 0x7f4880135800 2024-08-16T02:41:36.925 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.923+0000 7f487f7fe640 1 -- 172.21.15.133:0/3491515010 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f487800e230 con 0x7f4880135800 2024-08-16T02:41:36.925 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.923+0000 7f487f7fe640 1 -- 172.21.15.133:0/3491515010 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f487800e4d0 con 0x7f4880135800 2024-08-16T02:41:36.927 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.924+0000 7f488706d640 1 -- 172.21.15.133:0/3491515010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4880135800 msgr2=0x7f4880135c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.927 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.924+0000 7f488706d640 1 --2- 172.21.15.133:0/3491515010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4880135800 0x7f4880135c10 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.927 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.924+0000 7f488706d640 1 -- 172.21.15.133:0/3491515010 shutdown_connections 2024-08-16T02:41:36.927 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.924+0000 7f488706d640 1 --2- 172.21.15.133:0/3491515010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4880135800 0x7f4880135c10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.927 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.924+0000 7f488706d640 1 -- 172.21.15.133:0/3491515010 >> 172.21.15.133:0/3491515010 conn(0x7f488006caf0 msgr2=0x7f488006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:36.927 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.924+0000 7f488706d640 1 -- 172.21.15.133:0/3491515010 shutdown_connections 2024-08-16T02:41:36.927 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.924+0000 7f488706d640 1 -- 172.21.15.133:0/3491515010 wait complete. 2024-08-16T02:41:36.928 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.924+0000 7f488706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:36.928 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.925+0000 7f488706d640 1 Processor -- start 2024-08-16T02:41:36.928 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.925+0000 7f488706d640 1 -- start start 2024-08-16T02:41:36.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f488706d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f488009fe30 0x7f4880120be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f488706d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f4880132240 con 0x7f488009fe30 2024-08-16T02:41:36.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f4884e09640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f488009fe30 0x7f4880120be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f4884e09640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f488009fe30 0x7f4880120be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33528/0 (socket says 172.21.15.133:33528) 2024-08-16T02:41:36.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f4884e09640 1 -- 172.21.15.133:0/3729205422 learned_addr learned my addr 172.21.15.133:0/3729205422 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:36.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f4884e09640 1 -- 172.21.15.133:0/3729205422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488011f4a0 con 0x7f488009fe30 2024-08-16T02:41:36.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f4884e09640 1 --2- 172.21.15.133:0/3729205422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f488009fe30 0x7f4880120be0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4878014540 con 0x7f488009fe30 2024-08-16T02:41:36.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f48780146a0 con 0x7f488009fe30 2024-08-16T02:41:36.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488011ee50 con 0x7f488009fe30 2024-08-16T02:41:36.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4878014940 con 0x7f488009fe30 2024-08-16T02:41:36.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.927+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488011eb40 con 0x7f488009fe30 2024-08-16T02:41:36.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.928+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f48800728d0 con 0x7f488009fe30 2024-08-16T02:41:36.932 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.928+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f4878014af0 con 0x7f488009fe30 2024-08-16T02:41:36.932 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.929+0000 7f487d7fa640 1 --2- 172.21.15.133:0/3729205422 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f486003dd10 0x7f4860040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.932 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.929+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f4878019040 con 0x7f488009fe30 2024-08-16T02:41:36.932 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.929+0000 7f487d7fa640 1 --2- 172.21.15.133:0/3729205422 >> v2:172.21.15.133:6803/2214080839 conn(0x7f4860042d10 0x7f48600451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:36.932 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.929+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 --> v2:172.21.15.133:6803/2214080839 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4860045850 con 0x7f4860042d10 2024-08-16T02:41:36.932 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.929+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7f4878052460 con 0x7f488009fe30 2024-08-16T02:41:36.932 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.929+0000 7f487ffff640 1 --2- 172.21.15.133:0/3729205422 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f486003dd10 0x7f4860040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:36.933 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.929+0000 7f487ffff640 1 --2- 172.21.15.133:0/3729205422 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f486003dd10 0x7f4860040220 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.933 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.930+0000 7f488560a640 1 --2- 172.21.15.133:0/3729205422 >> v2:172.21.15.133:6803/2214080839 conn(0x7f4860042d10 0x7f48600451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:41:36.934 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.933+0000 7f488560a640 1 --2- 172.21.15.133:0/3729205422 >> v2:172.21.15.133:6803/2214080839 conn(0x7f4860042d10 0x7f48600451c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:36.935 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.934+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 <== osd.1 v2:172.21.15.133:6803/2214080839 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f4860045850 con 0x7f4860042d10 2024-08-16T02:41:36.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.938+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 --> v2:172.21.15.133:6803/2214080839 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4880074550 con 0x7f4860042d10 2024-08-16T02:41:36.940 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.939+0000 7f487d7fa640 1 -- 172.21.15.133:0/3729205422 <== osd.1 v2:172.21.15.133:6803/2214080839 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f4868009830 con 0x7f4860042d10 2024-08-16T02:41:36.940 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2024-08-16T02:41:36.940 INFO:teuthology.orchestra.run.smithi133.stdout: "success": "" 2024-08-16T02:41:36.941 INFO:teuthology.orchestra.run.smithi133.stdout:} 2024-08-16T02:41:36.941 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 >> v2:172.21.15.133:6803/2214080839 conn(0x7f4860042d10 msgr2=0x7f48600451c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 --2- 172.21.15.133:0/3729205422 >> v2:172.21.15.133:6803/2214080839 conn(0x7f4860042d10 0x7f48600451c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f486003dd10 msgr2=0x7f4860040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 --2- 172.21.15.133:0/3729205422 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f486003dd10 0x7f4860040220 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f488009fe30 msgr2=0x7f4880120be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:36.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 --2- 172.21.15.133:0/3729205422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f488009fe30 0x7f4880120be0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 shutdown_connections 2024-08-16T02:41:36.942 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 --2- 172.21.15.133:0/3729205422 >> v2:172.21.15.133:6803/2214080839 conn(0x7f4860042d10 0x7f48600451c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 --2- 172.21.15.133:0/3729205422 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f486003dd10 0x7f4860040220 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 --2- 172.21.15.133:0/3729205422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f488009fe30 0x7f4880120be0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:36.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 >> 172.21.15.133:0/3729205422 conn(0x7f488006caf0 msgr2=0x7f48801bd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:36.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 shutdown_connections 2024-08-16T02:41:36.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:36.940+0000 7f488706d640 1 -- 172.21.15.133:0/3729205422 wait complete. 2024-08-16T02:41:36.952 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-16T02:41:37.062 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.056+0000 7f5f21192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:37.062 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.062+0000 7f5f21192640 1 Processor -- start 2024-08-16T02:41:37.063 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.062+0000 7f5f21192640 1 -- start start 2024-08-16T02:41:37.064 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f21192640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5f1c09fe30 0x7f5f1c0a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.064 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f21192640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f1c0a1740 con 0x7f5f1c0a0780 2024-08-16T02:41:37.064 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f21192640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f5f1c06f380 con 0x7f5f1c09fe30 2024-08-16T02:41:37.065 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f1ad9d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5f1c09fe30 0x7f5f1c0a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.065 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f1a59c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f5f1c0a0780 0x7f5f1c135390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32994/0 (socket says 172.21.15.133:32994) 2024-08-16T02:41:37.065 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f1a59c640 1 -- 172.21.15.133:0/359309148 learned_addr learned my addr 172.21.15.133:0/359309148 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:37.065 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f1ad9d640 1 -- 172.21.15.133:0/359309148 >> v1:172.21.15.133:6789/0 conn(0x7f5f1c0a0780 legacy=0x7f5f1c135390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.065 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f1ad9d640 1 -- 172.21.15.133:0/359309148 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f1c06f690 con 0x7f5f1c09fe30 2024-08-16T02:41:37.065 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.063+0000 7f5f1ad9d640 1 --2- 172.21.15.133:0/359309148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c0a0240 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2726d0e7ae1f84e0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.065 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f19d9b640 1 -- 172.21.15.133:0/359309148 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5f0c009d80 con 0x7f5f1c09fe30 2024-08-16T02:41:37.065 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f19d9b640 1 -- 172.21.15.133:0/359309148 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5f0c00e230 con 0x7f5f1c09fe30 2024-08-16T02:41:37.066 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f19d9b640 1 -- 172.21.15.133:0/359309148 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5f0c00e4d0 con 0x7f5f1c09fe30 2024-08-16T02:41:37.067 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f21192640 1 -- 172.21.15.133:0/359309148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 msgr2=0x7f5f1c0a0240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.067 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f21192640 1 --2- 172.21.15.133:0/359309148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c0a0240 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.067 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f21192640 1 -- 172.21.15.133:0/359309148 shutdown_connections 2024-08-16T02:41:37.067 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f21192640 1 --2- 172.21.15.133:0/359309148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c0a0240 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.067 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f21192640 1 -- 172.21.15.133:0/359309148 >> 172.21.15.133:0/359309148 conn(0x7f5f1c06caf0 msgr2=0x7f5f1c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:37.068 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f21192640 1 -- 172.21.15.133:0/359309148 shutdown_connections 2024-08-16T02:41:37.068 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.064+0000 7f5f21192640 1 -- 172.21.15.133:0/359309148 wait complete. 2024-08-16T02:41:37.068 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.065+0000 7f5f21192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:37.068 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.066+0000 7f5f21192640 1 Processor -- start 2024-08-16T02:41:37.068 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.066+0000 7f5f21192640 1 -- start start 2024-08-16T02:41:37.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.067+0000 7f5f21192640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c123770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.067+0000 7f5f21192640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1c1251d0 con 0x7f5f1c09fe30 2024-08-16T02:41:37.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.067+0000 7f5f1ad9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.067+0000 7f5f1ad9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c123770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33546/0 (socket says 172.21.15.133:33546) 2024-08-16T02:41:37.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.067+0000 7f5f1ad9d640 1 -- 172.21.15.133:0/2206864341 learned_addr learned my addr 172.21.15.133:0/2206864341 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:37.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.068+0000 7f5f1ad9d640 1 -- 172.21.15.133:0/2206864341 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f1c1233b0 con 0x7f5f1c09fe30 2024-08-16T02:41:37.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.068+0000 7f5f1ad9d640 1 --2- 172.21.15.133:0/2206864341 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c123770 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.068+0000 7f5f037fe640 1 -- 172.21.15.133:0/2206864341 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5f0c014620 con 0x7f5f1c09fe30 2024-08-16T02:41:37.071 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.068+0000 7f5f037fe640 1 -- 172.21.15.133:0/2206864341 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5f0c014780 con 0x7f5f1c09fe30 2024-08-16T02:41:37.071 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.068+0000 7f5f037fe640 1 -- 172.21.15.133:0/2206864341 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5f0c014a20 con 0x7f5f1c09fe30 2024-08-16T02:41:37.071 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.068+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f1c122db0 con 0x7f5f1c09fe30 2024-08-16T02:41:37.071 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.068+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f1c1255d0 con 0x7f5f1c09fe30 2024-08-16T02:41:37.071 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.069+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f1c124fe0 con 0x7f5f1c09fe30 2024-08-16T02:41:37.072 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.069+0000 7f5f037fe640 1 -- 172.21.15.133:0/2206864341 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f5f0c014c20 con 0x7f5f1c09fe30 2024-08-16T02:41:37.072 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.070+0000 7f5f037fe640 1 --2- 172.21.15.133:0/2206864341 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5f0803dcc0 0x7f5f080401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.072 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.070+0000 7f5f037fe640 1 -- 172.21.15.133:0/2206864341 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f5f0c019040 con 0x7f5f1c09fe30 2024-08-16T02:41:37.072 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.070+0000 7f5f1a59c640 1 --2- 172.21.15.133:0/2206864341 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5f0803dcc0 0x7f5f080401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.072 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.070+0000 7f5f1a59c640 1 --2- 172.21.15.133:0/2206864341 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5f0803dcc0 0x7f5f080401d0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.075 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.075+0000 7f5f037fe640 1 -- 172.21.15.133:0/2206864341 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f5f0c018970 con 0x7f5f1c09fe30 2024-08-16T02:41:37.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.262+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f5f1c1231b0 con 0x7f5f0803dcc0 2024-08-16T02:41:37.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.263+0000 7f5f037fe640 1 -- 172.21.15.133:0/2206864341 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f5f1c1231b0 con 0x7f5f0803dcc0 2024-08-16T02:41:37.264 INFO:teuthology.orchestra.run.smithi133.stderr:instructed osd(s) 1 to deep-scrub 2024-08-16T02:41:37.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.265+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5f0803dcc0 msgr2=0x7f5f080401d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.265+0000 7f5f21192640 1 --2- 172.21.15.133:0/2206864341 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5f0803dcc0 0x7f5f080401d0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.265+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 msgr2=0x7f5f1c123770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.265+0000 7f5f21192640 1 --2- 172.21.15.133:0/2206864341 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c123770 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.266+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 shutdown_connections 2024-08-16T02:41:37.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.266+0000 7f5f21192640 1 --2- 172.21.15.133:0/2206864341 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f5f0803dcc0 0x7f5f080401d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.266+0000 7f5f21192640 1 --2- 172.21.15.133:0/2206864341 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f1c09fe30 0x7f5f1c123770 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.266+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 >> 172.21.15.133:0/2206864341 conn(0x7f5f1c06caf0 msgr2=0x7f5f1c071700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:37.268 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.266+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 shutdown_connections 2024-08-16T02:41:37.268 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.266+0000 7f5f21192640 1 -- 172.21.15.133:0/2206864341 wait complete. 2024-08-16T02:41:37.281 INFO:tasks.ceph:Scrubbing osd.2 2024-08-16T02:41:37.281 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:41:37.395 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.385+0000 7f29a5792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:37.396 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.395+0000 7f29a5792640 1 Processor -- start 2024-08-16T02:41:37.396 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.395+0000 7f29a5792640 1 -- start start 2024-08-16T02:41:37.398 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.396+0000 7f29a5792640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f29a01383b0 0x7f29a013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.398 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.396+0000 7f29a5792640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a00747a0 con 0x7f29a0135520 2024-08-16T02:41:37.398 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.396+0000 7f29a5792640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f29a006ff20 con 0x7f29a01383b0 2024-08-16T02:41:37.398 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.396+0000 7f299effd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f29a0135520 0x7f29a0135930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:32998/0 (socket says 172.21.15.133:32998) 2024-08-16T02:41:37.398 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.396+0000 7f299effd640 1 -- 172.21.15.133:0/1208405048 learned_addr learned my addr 172.21.15.133:0/1208405048 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:37.398 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.396+0000 7f299e7fc640 1 --2- 172.21.15.133:0/1208405048 >> v2:172.21.15.133:3300/0 conn(0x7f29a01383b0 0x7f29a013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.398 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.396+0000 7f299dffb640 1 -- 172.21.15.133:0/1208405048 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818787942 0 0) 0x7f29a00747a0 con 0x7f29a0135520 2024-08-16T02:41:37.399 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.396+0000 7f299dffb640 1 -- 172.21.15.133:0/1208405048 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f298c003580 con 0x7f29a0135520 2024-08-16T02:41:37.399 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.397+0000 7f299e7fc640 1 -- 172.21.15.133:0/1208405048 >> v1:172.21.15.133:6789/0 conn(0x7f29a0135520 legacy=0x7f29a0135930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.399 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.397+0000 7f299e7fc640 1 -- 172.21.15.133:0/1208405048 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a0071fb0 con 0x7f29a01383b0 2024-08-16T02:41:37.399 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.397+0000 7f299e7fc640 1 --2- 172.21.15.133:0/1208405048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a01383b0 0x7f29a013a7d0 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df1d18a862123f6 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.399 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.397+0000 7f299dffb640 1 -- 172.21.15.133:0/1208405048 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2988009d80 con 0x7f29a01383b0 2024-08-16T02:41:37.399 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.397+0000 7f299dffb640 1 -- 172.21.15.133:0/1208405048 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f298800e230 con 0x7f29a01383b0 2024-08-16T02:41:37.399 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.397+0000 7f299dffb640 1 -- 172.21.15.133:0/1208405048 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f298800e4d0 con 0x7f29a01383b0 2024-08-16T02:41:37.401 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.398+0000 7f29a5792640 1 -- 172.21.15.133:0/1208405048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a01383b0 msgr2=0x7f29a013a7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.401 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.398+0000 7f29a5792640 1 --2- 172.21.15.133:0/1208405048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a01383b0 0x7f29a013a7d0 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.401 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.398+0000 7f29a5792640 1 -- 172.21.15.133:0/1208405048 shutdown_connections 2024-08-16T02:41:37.401 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.398+0000 7f29a5792640 1 --2- 172.21.15.133:0/1208405048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a01383b0 0x7f29a013a7d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.401 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.398+0000 7f29a5792640 1 -- 172.21.15.133:0/1208405048 >> 172.21.15.133:0/1208405048 conn(0x7f29a00a63e0 msgr2=0x7f29a00a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:37.401 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.398+0000 7f29a5792640 1 -- 172.21.15.133:0/1208405048 shutdown_connections 2024-08-16T02:41:37.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.398+0000 7f29a5792640 1 -- 172.21.15.133:0/1208405048 wait complete. 2024-08-16T02:41:37.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.398+0000 7f29a5792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:37.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.399+0000 7f29a5792640 1 Processor -- start 2024-08-16T02:41:37.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.399+0000 7f29a5792640 1 -- start start 2024-08-16T02:41:37.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.400+0000 7f29a5792640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a0135520 0x7f29a01c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.400+0000 7f29a5792640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f29a011fe80 con 0x7f29a0135520 2024-08-16T02:41:37.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f299effd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a0135520 0x7f29a01c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f299effd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a0135520 0x7f29a01c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33568/0 (socket says 172.21.15.133:33568) 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f299effd640 1 -- 172.21.15.133:0/2367627921 learned_addr learned my addr 172.21.15.133:0/2367627921 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f299effd640 1 -- 172.21.15.133:0/2367627921 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a01214a0 con 0x7f29a0135520 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f299effd640 1 --2- 172.21.15.133:0/2367627921 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a0135520 0x7f29a01c9ce0 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f29980070b0 con 0x7f29a0135520 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2998007210 con 0x7f29a0135520 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a011fae0 con 0x7f29a0135520 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f29980074b0 con 0x7f29a0135520 2024-08-16T02:41:37.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.401+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a0122290 con 0x7f29a0135520 2024-08-16T02:41:37.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.402+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f29a0123880 con 0x7f29a0135520 2024-08-16T02:41:37.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.402+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f2998007610 con 0x7f29a0135520 2024-08-16T02:41:37.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.402+0000 7f29977fe640 1 --2- 172.21.15.133:0/2367627921 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f298003dd10 0x7f2980040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.403+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f299804f0b0 con 0x7f29a0135520 2024-08-16T02:41:37.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.403+0000 7f29977fe640 1 --2- 172.21.15.133:0/2367627921 >> v2:172.21.15.137:6808/3166527287 conn(0x7f2980042d10 0x7f29800451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.403+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 --> v2:172.21.15.137:6808/3166527287 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2980045850 con 0x7f2980042d10 2024-08-16T02:41:37.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.403+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7f29980515a0 con 0x7f29a0135520 2024-08-16T02:41:37.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.403+0000 7f299e7fc640 1 --2- 172.21.15.133:0/2367627921 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f298003dd10 0x7f2980040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.407 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.404+0000 7f299e7fc640 1 --2- 172.21.15.133:0/2367627921 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f298003dd10 0x7f2980040220 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.407 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.404+0000 7f299f7fe640 1 --2- 172.21.15.133:0/2367627921 >> v2:172.21.15.137:6808/3166527287 conn(0x7f2980042d10 0x7f29800451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:41:37.409 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.408+0000 7f299f7fe640 1 --2- 172.21.15.133:0/2367627921 >> v2:172.21.15.137:6808/3166527287 conn(0x7f2980042d10 0x7f29800451c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.411 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.410+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 <== osd.2 v2:172.21.15.137:6808/3166527287 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f2980045850 con 0x7f2980042d10 2024-08-16T02:41:37.414 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.413+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 --> v2:172.21.15.137:6808/3166527287 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f29a0138580 con 0x7f2980042d10 2024-08-16T02:41:37.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.415+0000 7f29977fe640 1 -- 172.21.15.133:0/2367627921 <== osd.2 v2:172.21.15.137:6808/3166527287 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f29a0138580 con 0x7f2980042d10 2024-08-16T02:41:37.416 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2024-08-16T02:41:37.417 INFO:teuthology.orchestra.run.smithi133.stdout: "success": "" 2024-08-16T02:41:37.417 INFO:teuthology.orchestra.run.smithi133.stdout:} 2024-08-16T02:41:37.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 >> v2:172.21.15.137:6808/3166527287 conn(0x7f2980042d10 msgr2=0x7f29800451c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 --2- 172.21.15.133:0/2367627921 >> v2:172.21.15.137:6808/3166527287 conn(0x7f2980042d10 0x7f29800451c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f298003dd10 msgr2=0x7f2980040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 --2- 172.21.15.133:0/2367627921 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f298003dd10 0x7f2980040220 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a0135520 msgr2=0x7f29a01c9ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 --2- 172.21.15.133:0/2367627921 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a0135520 0x7f29a01c9ce0 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 shutdown_connections 2024-08-16T02:41:37.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 --2- 172.21.15.133:0/2367627921 >> v2:172.21.15.137:6808/3166527287 conn(0x7f2980042d10 0x7f29800451c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 --2- 172.21.15.133:0/2367627921 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f298003dd10 0x7f2980040220 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 --2- 172.21.15.133:0/2367627921 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29a0135520 0x7f29a01c9ce0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 >> 172.21.15.133:0/2367627921 conn(0x7f29a00a63e0 msgr2=0x7f29a0072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:37.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 shutdown_connections 2024-08-16T02:41:37.419 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.416+0000 7f29a5792640 1 -- 172.21.15.133:0/2367627921 wait complete. 2024-08-16T02:41:37.427 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-16T02:41:37.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.530+0000 7f59192ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:37.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.536+0000 7f59192ad640 1 Processor -- start 2024-08-16T02:41:37.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.536+0000 7f59192ad640 1 -- start start 2024-08-16T02:41:37.539 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.537+0000 7f59192ad640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5914133ac0 0x7f59141384f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.539 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f59192ad640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5914071bd0 con 0x7f5914132fa0 2024-08-16T02:41:37.539 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f59192ad640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f5914073660 con 0x7f5914133ac0 2024-08-16T02:41:37.540 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f59127fc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5914133ac0 0x7f59141384f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.540 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f5912ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f5914132fa0 0x7f59141333b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:33006/0 (socket says 172.21.15.133:33006) 2024-08-16T02:41:37.540 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f59127fc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5914133ac0 0x7f59141384f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33584/0 (socket says 172.21.15.133:33584) 2024-08-16T02:41:37.540 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f5912ffd640 1 -- 172.21.15.133:0/2850190343 learned_addr learned my addr 172.21.15.133:0/2850190343 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:37.540 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f59127fc640 1 -- 172.21.15.133:0/2850190343 >> v1:172.21.15.133:6789/0 conn(0x7f5914132fa0 legacy=0x7f59141333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.540 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f59127fc640 1 -- 172.21.15.133:0/2850190343 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59140a4d90 con 0x7f5914133ac0 2024-08-16T02:41:37.540 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f5911ffb640 1 -- 172.21.15.133:0/2850190343 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736782589 0 0) 0x7f5914071bd0 con 0x7f5914132fa0 2024-08-16T02:41:37.540 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f59127fc640 1 --2- 172.21.15.133:0/2850190343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914133ac0 0x7f59141384f0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a735f490f200b4 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.541 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f5911ffb640 1 -- 172.21.15.133:0/2850190343 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f58fc009d80 con 0x7f5914133ac0 2024-08-16T02:41:37.541 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f5911ffb640 1 -- 172.21.15.133:0/2850190343 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f58fc00e230 con 0x7f5914133ac0 2024-08-16T02:41:37.541 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.538+0000 7f5911ffb640 1 -- 172.21.15.133:0/2850190343 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f58fc00e4d0 con 0x7f5914133ac0 2024-08-16T02:41:37.542 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.539+0000 7f59192ad640 1 -- 172.21.15.133:0/2850190343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914133ac0 msgr2=0x7f59141384f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.542 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.539+0000 7f59192ad640 1 --2- 172.21.15.133:0/2850190343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914133ac0 0x7f59141384f0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.542 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.539+0000 7f59192ad640 1 -- 172.21.15.133:0/2850190343 shutdown_connections 2024-08-16T02:41:37.542 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.539+0000 7f59192ad640 1 --2- 172.21.15.133:0/2850190343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914133ac0 0x7f59141384f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.542 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.539+0000 7f59192ad640 1 -- 172.21.15.133:0/2850190343 >> 172.21.15.133:0/2850190343 conn(0x7f59140a6300 msgr2=0x7f59140a2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:37.543 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.539+0000 7f59192ad640 1 -- 172.21.15.133:0/2850190343 shutdown_connections 2024-08-16T02:41:37.543 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.539+0000 7f59192ad640 1 -- 172.21.15.133:0/2850190343 wait complete. 2024-08-16T02:41:37.543 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.539+0000 7f59192ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:37.543 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.541+0000 7f59192ad640 1 Processor -- start 2024-08-16T02:41:37.543 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.541+0000 7f59192ad640 1 -- start start 2024-08-16T02:41:37.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.542+0000 7f59192ad640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914132fa0 0x7f591411fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.542+0000 7f59192ad640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f5914132e60 con 0x7f5914132fa0 2024-08-16T02:41:37.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.542+0000 7f5912ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914132fa0 0x7f591411fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.542+0000 7f5912ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914132fa0 0x7f591411fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33596/0 (socket says 172.21.15.133:33596) 2024-08-16T02:41:37.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.542+0000 7f5912ffd640 1 -- 172.21.15.133:0/2374273890 learned_addr learned my addr 172.21.15.133:0/2374273890 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:37.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.542+0000 7f5912ffd640 1 -- 172.21.15.133:0/2374273890 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5914121470 con 0x7f5914132fa0 2024-08-16T02:41:37.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.542+0000 7f5912ffd640 1 --2- 172.21.15.133:0/2374273890 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914132fa0 0x7f591411fa70 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.543+0000 7f590b7fe640 1 -- 172.21.15.133:0/2374273890 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5904004bc0 con 0x7f5914132fa0 2024-08-16T02:41:37.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.543+0000 7f590b7fe640 1 -- 172.21.15.133:0/2374273890 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5904004d20 con 0x7f5914132fa0 2024-08-16T02:41:37.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.543+0000 7f590b7fe640 1 -- 172.21.15.133:0/2374273890 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5904007120 con 0x7f5914132fa0 2024-08-16T02:41:37.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.543+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5914120c00 con 0x7f5914132fa0 2024-08-16T02:41:37.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.543+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5914122290 con 0x7f5914132fa0 2024-08-16T02:41:37.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.544+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f591406a020 con 0x7f5914132fa0 2024-08-16T02:41:37.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.544+0000 7f590b7fe640 1 -- 172.21.15.133:0/2374273890 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f59040072d0 con 0x7f5914132fa0 2024-08-16T02:41:37.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.544+0000 7f590b7fe640 1 --2- 172.21.15.133:0/2374273890 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f58f403dd10 0x7f58f4040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.544+0000 7f590b7fe640 1 -- 172.21.15.133:0/2374273890 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f590404ef80 con 0x7f5914132fa0 2024-08-16T02:41:37.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.544+0000 7f59127fc640 1 --2- 172.21.15.133:0/2374273890 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f58f403dd10 0x7f58f4040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.545+0000 7f59127fc640 1 --2- 172.21.15.133:0/2374273890 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f58f403dd10 0x7f58f4040220 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.550 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.549+0000 7f590b7fe640 1 -- 172.21.15.133:0/2374273890 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f5904006290 con 0x7f5914132fa0 2024-08-16T02:41:37.735 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.734+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f59140a48e0 con 0x7f58f403dd10 2024-08-16T02:41:37.735 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.735+0000 7f590b7fe640 1 -- 172.21.15.133:0/2374273890 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f58fc00e3a0 con 0x7f58f403dd10 2024-08-16T02:41:37.735 INFO:teuthology.orchestra.run.smithi133.stderr:instructed osd(s) 2 to deep-scrub 2024-08-16T02:41:37.738 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f58f403dd10 msgr2=0x7f58f4040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.738 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 --2- 172.21.15.133:0/2374273890 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f58f403dd10 0x7f58f4040220 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.738 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914132fa0 msgr2=0x7f591411fa70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.738 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 --2- 172.21.15.133:0/2374273890 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914132fa0 0x7f591411fa70 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.739 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 shutdown_connections 2024-08-16T02:41:37.739 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 --2- 172.21.15.133:0/2374273890 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f58f403dd10 0x7f58f4040220 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.739 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 --2- 172.21.15.133:0/2374273890 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5914132fa0 0x7f591411fa70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.739 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 >> 172.21.15.133:0/2374273890 conn(0x7f59140a6300 msgr2=0x7f591409c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:37.739 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 shutdown_connections 2024-08-16T02:41:37.739 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.737+0000 7f59192ad640 1 -- 172.21.15.133:0/2374273890 wait complete. 2024-08-16T02:41:37.752 INFO:tasks.ceph:Scrubbing osd.3 2024-08-16T02:41:37.752 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:41:37.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.856+0000 7f20c4592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:37.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.865+0000 7f20c4592640 1 Processor -- start 2024-08-16T02:41:37.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.865+0000 7f20c4592640 1 -- start start 2024-08-16T02:41:37.868 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.866+0000 7f20c4592640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f20bc071bd0 0x7f20bc071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.868 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.866+0000 7f20c4592640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20bc074240 con 0x7f20bc072520 2024-08-16T02:41:37.868 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.866+0000 7f20c4592640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f20bc074570 con 0x7f20bc071bd0 2024-08-16T02:41:37.868 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c232e640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f20bc071bd0 0x7f20bc071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c1b2d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f20bc072520 0x7f20bc132fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:33014/0 (socket says 172.21.15.133:33014) 2024-08-16T02:41:37.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c1b2d640 1 -- 172.21.15.133:0/3921003791 learned_addr learned my addr 172.21.15.133:0/3921003791 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:37.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c232e640 1 -- 172.21.15.133:0/3921003791 >> v1:172.21.15.133:6789/0 conn(0x7f20bc072520 legacy=0x7f20bc132fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c232e640 1 -- 172.21.15.133:0/3921003791 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc06c150 con 0x7f20bc071bd0 2024-08-16T02:41:37.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c232e640 1 --2- 172.21.15.133:0/3921003791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc071fe0 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1331aeaffe89e server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c132c640 1 -- 172.21.15.133:0/3921003791 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f20b8009d90 con 0x7f20bc071bd0 2024-08-16T02:41:37.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c132c640 1 -- 172.21.15.133:0/3921003791 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f20b800e230 con 0x7f20bc071bd0 2024-08-16T02:41:37.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.867+0000 7f20c132c640 1 -- 172.21.15.133:0/3921003791 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f20b800e4d0 con 0x7f20bc071bd0 2024-08-16T02:41:37.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.868+0000 7f20c4592640 1 -- 172.21.15.133:0/3921003791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 msgr2=0x7f20bc071fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.868+0000 7f20c4592640 1 --2- 172.21.15.133:0/3921003791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc071fe0 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.868+0000 7f20c4592640 1 -- 172.21.15.133:0/3921003791 shutdown_connections 2024-08-16T02:41:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.868+0000 7f20c4592640 1 --2- 172.21.15.133:0/3921003791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc071fe0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.868+0000 7f20c4592640 1 -- 172.21.15.133:0/3921003791 >> 172.21.15.133:0/3921003791 conn(0x7f20bc0a63e0 msgr2=0x7f20bc0a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.868+0000 7f20c4592640 1 -- 172.21.15.133:0/3921003791 shutdown_connections 2024-08-16T02:41:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.868+0000 7f20c4592640 1 -- 172.21.15.133:0/3921003791 wait complete. 2024-08-16T02:41:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.868+0000 7f20c4592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.870+0000 7f20c4592640 1 Processor -- start 2024-08-16T02:41:37.872 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.870+0000 7f20c4592640 1 -- start start 2024-08-16T02:41:37.874 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c4592640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc11fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.874 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c4592640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc1239d0 con 0x7f20bc071bd0 2024-08-16T02:41:37.874 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c232e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc11fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.874 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c232e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc11fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33612/0 (socket says 172.21.15.133:33612) 2024-08-16T02:41:37.874 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c232e640 1 -- 172.21.15.133:0/1582301549 learned_addr learned my addr 172.21.15.133:0/1582301549 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:37.874 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c232e640 1 -- 172.21.15.133:0/1582301549 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc121760 con 0x7f20bc071bd0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c232e640 1 --2- 172.21.15.133:0/1582301549 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc11fcb0 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f20b8014540 con 0x7f20bc071bd0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f20b80146a0 con 0x7f20bc071bd0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20bc120ef0 con 0x7f20bc071bd0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f20b8014940 con 0x7f20bc071bd0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.871+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20bc122580 con 0x7f20bc071bd0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.872+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f20bc123b40 con 0x7f20bc071bd0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.872+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f20b8014af0 con 0x7f20bc071bd0 2024-08-16T02:41:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.873+0000 7f20b2ffd640 1 --2- 172.21.15.133:0/1582301549 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f20a003dd10 0x7f20a0040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.876 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.873+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f20b8019040 con 0x7f20bc071bd0 2024-08-16T02:41:37.876 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.873+0000 7f20b2ffd640 1 --2- 172.21.15.133:0/1582301549 >> v2:172.21.15.137:6805/34553420 conn(0x7f20a0042d10 0x7f20a00451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:37.876 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.873+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 --> v2:172.21.15.137:6805/34553420 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20a0045850 con 0x7f20a0042d10 2024-08-16T02:41:37.876 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.873+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7f20b80523b0 con 0x7f20bc071bd0 2024-08-16T02:41:37.876 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.873+0000 7f20c1b2d640 1 --2- 172.21.15.133:0/1582301549 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f20a003dd10 0x7f20a0040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:37.877 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.873+0000 7f20c1b2d640 1 --2- 172.21.15.133:0/1582301549 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f20a003dd10 0x7f20a0040220 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.877 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.874+0000 7f20c2b2f640 1 --2- 172.21.15.133:0/1582301549 >> v2:172.21.15.137:6805/34553420 conn(0x7f20a0042d10 0x7f20a00451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:41:37.879 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.878+0000 7f20c2b2f640 1 --2- 172.21.15.133:0/1582301549 >> v2:172.21.15.137:6805/34553420 conn(0x7f20a0042d10 0x7f20a00451c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:37.881 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.880+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 <== osd.3 v2:172.21.15.137:6805/34553420 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f20a0045850 con 0x7f20a0042d10 2024-08-16T02:41:37.884 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.883+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 --> v2:172.21.15.137:6805/34553420 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f20bc09d520 con 0x7f20a0042d10 2024-08-16T02:41:37.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.885+0000 7f20b2ffd640 1 -- 172.21.15.133:0/1582301549 <== osd.3 v2:172.21.15.137:6805/34553420 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f20b400a820 con 0x7f20a0042d10 2024-08-16T02:41:37.887 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2024-08-16T02:41:37.887 INFO:teuthology.orchestra.run.smithi133.stdout: "success": "" 2024-08-16T02:41:37.887 INFO:teuthology.orchestra.run.smithi133.stdout:} 2024-08-16T02:41:37.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 >> v2:172.21.15.137:6805/34553420 conn(0x7f20a0042d10 msgr2=0x7f20a00451c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 --2- 172.21.15.133:0/1582301549 >> v2:172.21.15.137:6805/34553420 conn(0x7f20a0042d10 0x7f20a00451c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f20a003dd10 msgr2=0x7f20a0040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 --2- 172.21.15.133:0/1582301549 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f20a003dd10 0x7f20a0040220 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 msgr2=0x7f20bc11fcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 --2- 172.21.15.133:0/1582301549 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc11fcb0 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 shutdown_connections 2024-08-16T02:41:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 --2- 172.21.15.133:0/1582301549 >> v2:172.21.15.137:6805/34553420 conn(0x7f20a0042d10 0x7f20a00451c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 --2- 172.21.15.133:0/1582301549 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f20a003dd10 0x7f20a0040220 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 --2- 172.21.15.133:0/1582301549 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f20bc071bd0 0x7f20bc11fcb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 >> 172.21.15.133:0/1582301549 conn(0x7f20bc0a63e0 msgr2=0x7f20bc0a4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:37.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 shutdown_connections 2024-08-16T02:41:37.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:37.886+0000 7f20c4592640 1 -- 172.21.15.133:0/1582301549 wait complete. 2024-08-16T02:41:37.897 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-08-16T02:41:38.010 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.005+0000 7f2c0562d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:38.011 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.011+0000 7f2c0562d640 1 Processor -- start 2024-08-16T02:41:38.011 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.011+0000 7f2c0562d640 1 -- start start 2024-08-16T02:41:38.013 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2c0562d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f2c000a0950 0x7f2c0013a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:38.013 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2c0562d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c000a1740 con 0x7f2c0009fe30 2024-08-16T02:41:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2c0562d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f2c0006f380 con 0x7f2c000a0950 2024-08-16T02:41:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2bfe7fc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f2c000a0950 0x7f2c0013a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2bfeffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2c0009fe30 0x7f2c000a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:33026/0 (socket says 172.21.15.133:33026) 2024-08-16T02:41:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2bfeffd640 1 -- 172.21.15.133:0/1780287719 learned_addr learned my addr 172.21.15.133:0/1780287719 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2bfe7fc640 1 -- 172.21.15.133:0/1780287719 >> v1:172.21.15.133:6789/0 conn(0x7f2c0009fe30 legacy=0x7f2c000a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2bfe7fc640 1 -- 172.21.15.133:0/1780287719 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0006f690 con 0x7f2c000a0950 2024-08-16T02:41:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2bfdffb640 1 -- 172.21.15.133:0/1780287719 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2832924177 0 0) 0x7f2c000a1740 con 0x7f2c0009fe30 2024-08-16T02:41:38.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.012+0000 7f2bfe7fc640 1 --2- 172.21.15.133:0/1780287719 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c000a0950 0x7f2c0013a640 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37d3663f50ab0cd server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:38.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2bfdffb640 1 -- 172.21.15.133:0/1780287719 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2bf4009d80 con 0x7f2c000a0950 2024-08-16T02:41:38.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2bfdffb640 1 -- 172.21.15.133:0/1780287719 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2bf400e230 con 0x7f2c000a0950 2024-08-16T02:41:38.015 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2bfdffb640 1 -- 172.21.15.133:0/1780287719 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2bf400e4d0 con 0x7f2c000a0950 2024-08-16T02:41:38.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2c0562d640 1 -- 172.21.15.133:0/1780287719 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c000a0950 msgr2=0x7f2c0013a640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:38.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2c0562d640 1 --2- 172.21.15.133:0/1780287719 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c000a0950 0x7f2c0013a640 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2c0562d640 1 -- 172.21.15.133:0/1780287719 shutdown_connections 2024-08-16T02:41:38.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2c0562d640 1 --2- 172.21.15.133:0/1780287719 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c000a0950 0x7f2c0013a640 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2c0562d640 1 -- 172.21.15.133:0/1780287719 >> 172.21.15.133:0/1780287719 conn(0x7f2c0006caf0 msgr2=0x7f2c0006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:38.016 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2c0562d640 1 -- 172.21.15.133:0/1780287719 shutdown_connections 2024-08-16T02:41:38.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.013+0000 7f2c0562d640 1 -- 172.21.15.133:0/1780287719 wait complete. 2024-08-16T02:41:38.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.014+0000 7f2c0562d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:38.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.015+0000 7f2c0562d640 1 Processor -- start 2024-08-16T02:41:38.017 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.015+0000 7f2c0562d640 1 -- start start 2024-08-16T02:41:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.016+0000 7f2c0562d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c0009fe30 0x7f2c001237a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.016+0000 7f2c0562d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f2c001233e0 con 0x7f2c0009fe30 2024-08-16T02:41:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.016+0000 7f2bfeffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c0009fe30 0x7f2c001237a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.016+0000 7f2bfeffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c0009fe30 0x7f2c001237a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33636/0 (socket says 172.21.15.133:33636) 2024-08-16T02:41:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.016+0000 7f2bfeffd640 1 -- 172.21.15.133:0/1404108103 learned_addr learned my addr 172.21.15.133:0/1404108103 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.016+0000 7f2bfeffd640 1 -- 172.21.15.133:0/1404108103 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c001247c0 con 0x7f2c0009fe30 2024-08-16T02:41:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.017+0000 7f2bfeffd640 1 --2- 172.21.15.133:0/1404108103 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c0009fe30 0x7f2c001237a0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.017+0000 7f2be77fe640 1 -- 172.21.15.133:0/1404108103 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2bf0004bc0 con 0x7f2c0009fe30 2024-08-16T02:41:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.017+0000 7f2be77fe640 1 -- 172.21.15.133:0/1404108103 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2bf0004d20 con 0x7f2c0009fe30 2024-08-16T02:41:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.017+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c00122de0 con 0x7f2c0009fe30 2024-08-16T02:41:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.017+0000 7f2be77fe640 1 -- 172.21.15.133:0/1404108103 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2bf0007120 con 0x7f2c0009fe30 2024-08-16T02:41:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.017+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c00124fe0 con 0x7f2c0009fe30 2024-08-16T02:41:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.018+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c000a0240 con 0x7f2c0009fe30 2024-08-16T02:41:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.018+0000 7f2be77fe640 1 -- 172.21.15.133:0/1404108103 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f2bf00072d0 con 0x7f2c0009fe30 2024-08-16T02:41:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.018+0000 7f2be77fe640 1 --2- 172.21.15.133:0/1404108103 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f2bec03dd10 0x7f2bec040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:38.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.018+0000 7f2be77fe640 1 -- 172.21.15.133:0/1404108103 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f2bf004eff0 con 0x7f2c0009fe30 2024-08-16T02:41:38.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.019+0000 7f2bfe7fc640 1 --2- 172.21.15.133:0/1404108103 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f2bec03dd10 0x7f2bec040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:38.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.019+0000 7f2bfe7fc640 1 --2- 172.21.15.133:0/1404108103 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f2bec03dd10 0x7f2bec040220 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:38.024 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.023+0000 7f2be77fe640 1 -- 172.21.15.133:0/1404108103 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f2bf0006290 con 0x7f2c0009fe30 2024-08-16T02:41:38.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.206+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f2c000a1410 con 0x7f2bec03dd10 2024-08-16T02:41:38.207 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.206+0000 7f2be77fe640 1 -- 172.21.15.133:0/1404108103 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f2bf4009d50 con 0x7f2bec03dd10 2024-08-16T02:41:38.207 INFO:teuthology.orchestra.run.smithi133.stderr:instructed osd(s) 3 to deep-scrub 2024-08-16T02:41:38.210 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f2bec03dd10 msgr2=0x7f2bec040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:38.210 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 --2- 172.21.15.133:0/1404108103 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f2bec03dd10 0x7f2bec040220 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.210 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c0009fe30 msgr2=0x7f2c001237a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:38.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 --2- 172.21.15.133:0/1404108103 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c0009fe30 0x7f2c001237a0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 shutdown_connections 2024-08-16T02:41:38.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 --2- 172.21.15.133:0/1404108103 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f2bec03dd10 0x7f2bec040220 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 --2- 172.21.15.133:0/1404108103 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c0009fe30 0x7f2c001237a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 >> 172.21.15.133:0/1404108103 conn(0x7f2c0006caf0 msgr2=0x7f2c00071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:38.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 shutdown_connections 2024-08-16T02:41:38.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.209+0000 7f2c0562d640 1 -- 172.21.15.133:0/1404108103 wait complete. 2024-08-16T02:41:38.222 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:41:38.331 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.326+0000 7f45a462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:38.331 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.331+0000 7f45a462d640 1 Processor -- start 2024-08-16T02:41:38.332 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.331+0000 7f45a462d640 1 -- start start 2024-08-16T02:41:38.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a462d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f459c09d650 0x7f459c137530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:38.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a462d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f459c0a0260 con 0x7f459c09cb30 2024-08-16T02:41:38.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a462d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f459c070900 con 0x7f459c09d650 2024-08-16T02:41:38.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a1bc8640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f459c09d650 0x7f459c137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:38.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a23c9640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f459c09cb30 0x7f459c09cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:33032/0 (socket says 172.21.15.133:33032) 2024-08-16T02:41:38.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a23c9640 1 -- 172.21.15.133:0/1004563358 learned_addr learned my addr 172.21.15.133:0/1004563358 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:38.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a1bc8640 1 -- 172.21.15.133:0/1004563358 >> v1:172.21.15.133:6789/0 conn(0x7f459c09cb30 legacy=0x7f459c09cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:38.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a1bc8640 1 -- 172.21.15.133:0/1004563358 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459c06f710 con 0x7f459c09d650 2024-08-16T02:41:38.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.332+0000 7f45a13c7640 1 -- 172.21.15.133:0/1004563358 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252958879 0 0) 0x7f459c0a0260 con 0x7f459c09cb30 2024-08-16T02:41:38.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.333+0000 7f45a1bc8640 1 --2- 172.21.15.133:0/1004563358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09d650 0x7f459c137530 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ceede0138c20c2 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:38.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.333+0000 7f45a13c7640 1 -- 172.21.15.133:0/1004563358 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4598009d90 con 0x7f459c09d650 2024-08-16T02:41:38.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.333+0000 7f45a13c7640 1 -- 172.21.15.133:0/1004563358 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f459800e230 con 0x7f459c09d650 2024-08-16T02:41:38.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.333+0000 7f45a13c7640 1 -- 172.21.15.133:0/1004563358 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f459800e4d0 con 0x7f459c09d650 2024-08-16T02:41:38.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.333+0000 7f45a462d640 1 -- 172.21.15.133:0/1004563358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09d650 msgr2=0x7f459c137530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:38.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.333+0000 7f45a462d640 1 --2- 172.21.15.133:0/1004563358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09d650 0x7f459c137530 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.334+0000 7f45a462d640 1 -- 172.21.15.133:0/1004563358 shutdown_connections 2024-08-16T02:41:38.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.334+0000 7f45a462d640 1 --2- 172.21.15.133:0/1004563358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09d650 0x7f459c137530 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.334+0000 7f45a462d640 1 -- 172.21.15.133:0/1004563358 >> 172.21.15.133:0/1004563358 conn(0x7f459c0a09c0 msgr2=0x7f459c0a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:38.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.334+0000 7f45a462d640 1 -- 172.21.15.133:0/1004563358 shutdown_connections 2024-08-16T02:41:38.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.334+0000 7f45a462d640 1 -- 172.21.15.133:0/1004563358 wait complete. 2024-08-16T02:41:38.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.334+0000 7f45a462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:38.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.335+0000 7f45a462d640 1 Processor -- start 2024-08-16T02:41:38.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.335+0000 7f45a462d640 1 -- start start 2024-08-16T02:41:38.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.336+0000 7f45a462d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09cb30 0x7f459c1c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:38.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.336+0000 7f45a462d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f459c1218f0 con 0x7f459c09cb30 2024-08-16T02:41:38.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.336+0000 7f45a23c9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09cb30 0x7f459c1c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:38.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.336+0000 7f45a23c9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09cb30 0x7f459c1c6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33650/0 (socket says 172.21.15.133:33650) 2024-08-16T02:41:38.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.336+0000 7f45a23c9640 1 -- 172.21.15.133:0/2266976901 learned_addr learned my addr 172.21.15.133:0/2266976901 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:38.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.337+0000 7f45a23c9640 1 -- 172.21.15.133:0/2266976901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459c120000 con 0x7f459c09cb30 2024-08-16T02:41:38.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.337+0000 7f45a23c9640 1 --2- 172.21.15.133:0/2266976901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09cb30 0x7f459c1c6900 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:38.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.337+0000 7f458effd640 1 -- 172.21.15.133:0/2266976901 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4590004b80 con 0x7f459c09cb30 2024-08-16T02:41:38.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.337+0000 7f458effd640 1 -- 172.21.15.133:0/2266976901 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4590004ce0 con 0x7f459c09cb30 2024-08-16T02:41:38.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.337+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459c1202a0 con 0x7f459c09cb30 2024-08-16T02:41:38.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.337+0000 7f458effd640 1 -- 172.21.15.133:0/2266976901 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4590007120 con 0x7f459c09cb30 2024-08-16T02:41:38.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.337+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f459c1221a0 con 0x7f459c09cb30 2024-08-16T02:41:38.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.338+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f459c09cf40 con 0x7f459c09cb30 2024-08-16T02:41:38.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.338+0000 7f458effd640 1 -- 172.21.15.133:0/2266976901 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f45900072d0 con 0x7f459c09cb30 2024-08-16T02:41:38.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.338+0000 7f458effd640 1 --2- 172.21.15.133:0/2266976901 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f458003dd10 0x7f4580040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:38.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.338+0000 7f458effd640 1 -- 172.21.15.133:0/2266976901 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f459004e880 con 0x7f459c09cb30 2024-08-16T02:41:38.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.339+0000 7f45a1bc8640 1 --2- 172.21.15.133:0/2266976901 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f458003dd10 0x7f4580040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:38.342 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.339+0000 7f45a1bc8640 1 --2- 172.21.15.133:0/2266976901 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f458003dd10 0x7f4580040220 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:38.344 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.343+0000 7f458effd640 1 -- 172.21.15.133:0/2266976901 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f459000adb0 con 0x7f459c09cb30 2024-08-16T02:41:38.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.525+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f459c09ff50 con 0x7f458003dd10 2024-08-16T02:41:38.529 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.527+0000 7f458effd640 1 -- 172.21.15.133:0/2266976901 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7f4598009e00 con 0x7f458003dd10 2024-08-16T02:41:38.530 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:41:38.531 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-08-16T02:41:38.531 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f458003dd10 msgr2=0x7f4580040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:38.531 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 --2- 172.21.15.133:0/2266976901 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f458003dd10 0x7f4580040220 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.531 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09cb30 msgr2=0x7f459c1c6900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:38.532 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 --2- 172.21.15.133:0/2266976901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09cb30 0x7f459c1c6900 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.532 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 shutdown_connections 2024-08-16T02:41:38.532 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 --2- 172.21.15.133:0/2266976901 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f458003dd10 0x7f4580040220 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.532 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 --2- 172.21.15.133:0/2266976901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f459c09cb30 0x7f459c1c6900 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:38.532 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 >> 172.21.15.133:0/2266976901 conn(0x7f459c0a09c0 msgr2=0x7f459c072690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:38.532 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 shutdown_connections 2024-08-16T02:41:38.532 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:38.529+0000 7f45a462d640 1 -- 172.21.15.133:0/2266976901 wait complete. 2024-08-16T02:41:38.541 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-08-16T02:41:37.441870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":12884,"kb_used_data":12884,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209178028,"statfs":{"total":214211493888,"available":214198300672,"internally_reserved":0,"allocated":13193216,"data_stored":13193216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.521229"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:32.973605+0000","last_change":"2024-08-16T02:36:32.491674+0000","last_active":"2024-08-16T02:41:32.973605+0000","last_peered":"2024-08-16T02:41:32.973605+0000","last_clean":"2024-08-16T02:41:32.973605+0000","last_became_active":"2024-08-16T02:36:30.530297+0000","last_became_peered":"2024-08-16T02:36:30.530297+0000","last_unstale":"2024-08-16T02:41:32.973605+0000","last_undegraded":"2024-08-16T02:41:32.973605+0000","last_fullsized":"2024-08-16T02:41:32.973605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:32.996175+0000","last_change":"2024-08-16T02:36:32.493152+0000","last_active":"2024-08-16T02:41:32.996175+0000","last_peered":"2024-08-16T02:41:32.996175+0000","last_clean":"2024-08-16T02:41:32.996175+0000","last_became_active":"2024-08-16T02:36:30.537306+0000","last_became_peered":"2024-08-16T02:36:30.537306+0000","last_unstale":"2024-08-16T02:41:32.996175+0000","last_undegraded":"2024-08-16T02:41:32.996175+0000","last_fullsized":"2024-08-16T02:41:32.996175+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:33.024914+0000","last_change":"2024-08-16T02:36:32.491067+0000","last_active":"2024-08-16T02:41:33.024914+0000","last_peered":"2024-08-16T02:41:33.024914+0000","last_clean":"2024-08-16T02:41:33.024914+0000","last_became_active":"2024-08-16T02:36:30.537547+0000","last_became_peered":"2024-08-16T02:36:30.537547+0000","last_unstale":"2024-08-16T02:41:33.024914+0000","last_undegraded":"2024-08-16T02:41:33.024914+0000","last_fullsized":"2024-08-16T02:41:33.024914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:32.970456+0000","last_change":"2024-08-16T02:36:32.489861+0000","last_active":"2024-08-16T02:41:32.970456+0000","last_peered":"2024-08-16T02:41:32.970456+0000","last_clean":"2024-08-16T02:41:32.970456+0000","last_became_active":"2024-08-16T02:36:30.521009+0000","last_became_peered":"2024-08-16T02:36:30.521009+0000","last_unstale":"2024-08-16T02:41:32.970456+0000","last_undegraded":"2024-08-16T02:41:32.970456+0000","last_fullsized":"2024-08-16T02:41:32.970456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:32.985790+0000","last_change":"2024-08-16T02:36:32.489707+0000","last_active":"2024-08-16T02:41:32.985790+0000","last_peered":"2024-08-16T02:41:32.985790+0000","last_clean":"2024-08-16T02:41:32.985790+0000","last_became_active":"2024-08-16T02:36:30.539438+0000","last_became_peered":"2024-08-16T02:36:30.539438+0000","last_unstale":"2024-08-16T02:41:32.985790+0000","last_undegraded":"2024-08-16T02:41:32.985790+0000","last_fullsized":"2024-08-16T02:41:32.985790+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:32.978281+0000","last_change":"2024-08-16T02:36:32.491302+0000","last_active":"2024-08-16T02:41:32.978281+0000","last_peered":"2024-08-16T02:41:32.978281+0000","last_clean":"2024-08-16T02:41:32.978281+0000","last_became_active":"2024-08-16T02:36:30.521005+0000","last_became_peered":"2024-08-16T02:36:30.521005+0000","last_unstale":"2024-08-16T02:41:32.978281+0000","last_undegraded":"2024-08-16T02:41:32.978281+0000","last_fullsized":"2024-08-16T02:41:32.978281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:33.011613+0000","last_change":"2024-08-16T02:36:32.490191+0000","last_active":"2024-08-16T02:41:33.011613+0000","last_peered":"2024-08-16T02:41:33.011613+0000","last_clean":"2024-08-16T02:41:33.011613+0000","last_became_active":"2024-08-16T02:36:30.548723+0000","last_became_peered":"2024-08-16T02:36:30.548723+0000","last_unstale":"2024-08-16T02:41:33.011613+0000","last_undegraded":"2024-08-16T02:41:33.011613+0000","last_fullsized":"2024-08-16T02:41:33.011613+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":655,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:33.006508+0000","last_change":"2024-08-16T02:36:32.495617+0000","last_active":"2024-08-16T02:41:33.006508+0000","last_peered":"2024-08-16T02:41:33.006508+0000","last_clean":"2024-08-16T02:41:33.006508+0000","last_became_active":"2024-08-16T02:36:30.528787+0000","last_became_peered":"2024-08-16T02:36:30.528787+0000","last_unstale":"2024-08-16T02:41:33.006508+0000","last_undegraded":"2024-08-16T02:41:33.006508+0000","last_fullsized":"2024-08-16T02:41:33.006508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:29.440942+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":726,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:32.982363+0000","last_change":"2024-08-16T02:36:28.507195+0000","last_active":"2024-08-16T02:41:32.982363+0000","last_peered":"2024-08-16T02:41:32.982363+0000","last_clean":"2024-08-16T02:41:32.982363+0000","last_became_active":"2024-08-16T02:36:28.489045+0000","last_became_peered":"2024-08-16T02:36:28.489045+0000","last_unstale":"2024-08-16T02:41:32.982363+0000","last_undegraded":"2024-08-16T02:41:32.982363+0000","last_fullsized":"2024-08-16T02:41:32.982363+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:36:27.432441+0000","last_clean_scrub_stamp":"2024-08-16T02:36:27.432441+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198300672,"internally_reserved":0,"allocated":13193216,"data_stored":13193216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3472,"kb_used_data":3472,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294256,"statfs":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2836,"kb_used_data":2836,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294892,"statfs":{"total":53552873472,"available":53549969408,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2828,"kb_used_data":2828,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294900,"statfs":{"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3748,"kb_used_data":3748,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293980,"statfs":{"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53549969408,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:41:38.544 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-16T02:36:29.440942+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=29, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.544 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-16T02:36:29.440942+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=29, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.544 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-16T02:36:29.440942+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=29, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.544 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-16T02:36:29.440942+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=29, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.544 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-16T02:36:29.440942+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=29, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.545 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-16T02:36:29.440942+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=29, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.545 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-16T02:36:29.440942+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=29, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.545 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-16T02:36:29.440942+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=29, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.545 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-16T02:36:27.432441+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=36, tm_sec=27, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=41, tm_sec=35, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:41:38.545 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-16T02:41:58.546 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:41:58.666 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.661+0000 7f252faed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:58.667 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.666+0000 7f252faed640 1 Processor -- start 2024-08-16T02:41:58.667 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.666+0000 7f252faed640 1 -- start start 2024-08-16T02:41:58.669 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.667+0000 7f252faed640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f252809cb30 0x7f252809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:58.669 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.667+0000 7f252faed640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25280a0260 con 0x7f252809d480 2024-08-16T02:41:58.669 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.667+0000 7f252faed640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7f2528070900 con 0x7f252809cb30 2024-08-16T02:41:58.669 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.667+0000 7f252d889640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f252809cb30 0x7f252809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:58.670 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.667+0000 7f252d088640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f252809d480 0x7f25281322f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:57006/0 (socket says 172.21.15.133:57006) 2024-08-16T02:41:58.670 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.667+0000 7f252d088640 1 -- 172.21.15.133:0/2876416250 learned_addr learned my addr 172.21.15.133:0/2876416250 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:58.670 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.668+0000 7f252d889640 1 -- 172.21.15.133:0/2876416250 >> v1:172.21.15.133:6789/0 conn(0x7f252809d480 legacy=0x7f25281322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:58.670 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.668+0000 7f252c887640 1 -- 172.21.15.133:0/2876416250 <== mon.0 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 386768510 0 0) 0x7f25280a0260 con 0x7f252809d480 2024-08-16T02:41:58.670 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.668+0000 7f252d889640 1 -- 172.21.15.133:0/2876416250 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252806f710 con 0x7f252809cb30 2024-08-16T02:41:58.670 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.668+0000 7f252d889640 1 --2- 172.21.15.133:0/2876416250 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f252809cf40 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cf319908464d5c server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:58.670 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.668+0000 7f252c887640 1 -- 172.21.15.133:0/2876416250 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2524009d90 con 0x7f252809cb30 2024-08-16T02:41:58.670 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.668+0000 7f252c887640 1 -- 172.21.15.133:0/2876416250 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f252400e230 con 0x7f252809cb30 2024-08-16T02:41:58.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.668+0000 7f252c887640 1 -- 172.21.15.133:0/2876416250 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f252400e4d0 con 0x7f252809cb30 2024-08-16T02:41:58.672 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.669+0000 7f252faed640 1 -- 172.21.15.133:0/2876416250 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 msgr2=0x7f252809cf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:58.672 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.669+0000 7f252faed640 1 --2- 172.21.15.133:0/2876416250 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f252809cf40 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:58.672 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.669+0000 7f252faed640 1 -- 172.21.15.133:0/2876416250 shutdown_connections 2024-08-16T02:41:58.672 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.669+0000 7f252faed640 1 --2- 172.21.15.133:0/2876416250 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f252809cf40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:58.672 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.669+0000 7f252faed640 1 -- 172.21.15.133:0/2876416250 >> 172.21.15.133:0/2876416250 conn(0x7f25280a09c0 msgr2=0x7f25280a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:58.672 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.669+0000 7f252faed640 1 -- 172.21.15.133:0/2876416250 shutdown_connections 2024-08-16T02:41:58.673 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.669+0000 7f252faed640 1 -- 172.21.15.133:0/2876416250 wait complete. 2024-08-16T02:41:58.673 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.669+0000 7f252faed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:58.673 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.670+0000 7f252faed640 1 Processor -- start 2024-08-16T02:41:58.673 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.671+0000 7f252faed640 1 -- start start 2024-08-16T02:41:58.675 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252faed640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f25281c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:58.675 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252faed640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7f2528123e60 con 0x7f252809cb30 2024-08-16T02:41:58.675 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252d889640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f25281c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:58.675 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252d889640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f25281c6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:42430/0 (socket says 172.21.15.133:42430) 2024-08-16T02:41:58.675 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252d889640 1 -- 172.21.15.133:0/555326578 learned_addr learned my addr 172.21.15.133:0/555326578 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:58.675 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252d889640 1 -- 172.21.15.133:0/555326578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252811fef0 con 0x7f252809cb30 2024-08-16T02:41:58.676 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252d889640 1 --2- 172.21.15.133:0/555326578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f25281c6900 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:58.676 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f251e7fc640 1 -- 172.21.15.133:0/555326578 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2524017040 con 0x7f252809cb30 2024-08-16T02:41:58.676 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f251e7fc640 1 -- 172.21.15.133:0/555326578 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2524012730 con 0x7f252809cb30 2024-08-16T02:41:58.676 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f251e7fc640 1 -- 172.21.15.133:0/555326578 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f25240129d0 con 0x7f252809cb30 2024-08-16T02:41:58.676 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2528122920 con 0x7f252809cb30 2024-08-16T02:41:58.676 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.672+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2528120190 con 0x7f252809cb30 2024-08-16T02:41:58.676 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.673+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25280008d0 con 0x7f252809cb30 2024-08-16T02:41:58.676 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.673+0000 7f251e7fc640 1 -- 172.21.15.133:0/555326578 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f252401a040 con 0x7f252809cb30 2024-08-16T02:41:58.677 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.674+0000 7f251e7fc640 1 --2- 172.21.15.133:0/555326578 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f250c03dd10 0x7f250c040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:58.677 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.674+0000 7f251e7fc640 1 -- 172.21.15.133:0/555326578 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f25240519a0 con 0x7f252809cb30 2024-08-16T02:41:58.677 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.674+0000 7f252d088640 1 --2- 172.21.15.133:0/555326578 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f250c03dd10 0x7f250c040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:58.677 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.675+0000 7f252d088640 1 --2- 172.21.15.133:0/555326578 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f250c03dd10 0x7f250c040220 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:58.679 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.679+0000 7f251e7fc640 1 -- 172.21.15.133:0/555326578 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f252400d1c0 con 0x7f252809cb30 2024-08-16T02:41:58.867 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.866+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 --> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f252809ff50 con 0x7f250c03dd10 2024-08-16T02:41:58.868 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.867+0000 7f251e7fc640 1 -- 172.21.15.133:0/555326578 <== mgr.4099 v2:172.21.15.137:6800/545649490 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31243 (crc 0 0 0) 0x7f2518005930 con 0x7f250c03dd10 2024-08-16T02:41:58.868 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-08-16T02:41:58.869 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-08-16T02:41:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f250c03dd10 msgr2=0x7f250c040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 --2- 172.21.15.133:0/555326578 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f250c03dd10 0x7f250c040220 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 msgr2=0x7f25281c6900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 --2- 172.21.15.133:0/555326578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f25281c6900 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 shutdown_connections 2024-08-16T02:41:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 --2- 172.21.15.133:0/555326578 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7f250c03dd10 0x7f250c040220 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 --2- 172.21.15.133:0/555326578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f252809cb30 0x7f25281c6900 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:58.872 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 >> 172.21.15.133:0/555326578 conn(0x7f25280a09c0 msgr2=0x7f2528072620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:58.872 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 shutdown_connections 2024-08-16T02:41:58.872 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.869+0000 7f252faed640 1 -- 172.21.15.133:0/555326578 wait complete. 2024-08-16T02:41:58.881 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-08-16T02:41:57.446810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":12884,"kb_used_data":12884,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209178028,"statfs":{"total":214211493888,"available":214198300672,"internally_reserved":0,"allocated":13193216,"data_stored":13193216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003078"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:37.267449+0000","last_change":"2024-08-16T02:41:37.284916+0000","last_active":"2024-08-16T02:41:37.267449+0000","last_peered":"2024-08-16T02:41:37.267449+0000","last_clean":"2024-08-16T02:41:37.267449+0000","last_became_active":"2024-08-16T02:36:30.530297+0000","last_became_peered":"2024-08-16T02:36:30.530297+0000","last_unstale":"2024-08-16T02:41:37.267449+0000","last_undegraded":"2024-08-16T02:41:37.267449+0000","last_fullsized":"2024-08-16T02:41:37.267449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:41:37.284742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:37.284742+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:37.739961+0000","last_change":"2024-08-16T02:41:37.754814+0000","last_active":"2024-08-16T02:41:37.739961+0000","last_peered":"2024-08-16T02:41:37.739961+0000","last_clean":"2024-08-16T02:41:37.739961+0000","last_became_active":"2024-08-16T02:36:30.537306+0000","last_became_peered":"2024-08-16T02:36:30.537306+0000","last_unstale":"2024-08-16T02:41:37.739961+0000","last_undegraded":"2024-08-16T02:41:37.739961+0000","last_fullsized":"2024-08-16T02:41:37.739961+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:41:37.754627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:37.754627+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:38.212254+0000","last_change":"2024-08-16T02:41:38.237474+0000","last_active":"2024-08-16T02:41:38.212254+0000","last_peered":"2024-08-16T02:41:38.212254+0000","last_clean":"2024-08-16T02:41:38.212254+0000","last_became_active":"2024-08-16T02:36:30.537547+0000","last_became_peered":"2024-08-16T02:36:30.537547+0000","last_unstale":"2024-08-16T02:41:38.212254+0000","last_undegraded":"2024-08-16T02:41:38.212254+0000","last_fullsized":"2024-08-16T02:41:38.212254+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:41:38.237269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:38.237269+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:37.737996+0000","last_change":"2024-08-16T02:41:37.752690+0000","last_active":"2024-08-16T02:41:37.737996+0000","last_peered":"2024-08-16T02:41:37.737996+0000","last_clean":"2024-08-16T02:41:37.737996+0000","last_became_active":"2024-08-16T02:36:30.521009+0000","last_became_peered":"2024-08-16T02:36:30.521009+0000","last_unstale":"2024-08-16T02:41:37.737996+0000","last_undegraded":"2024-08-16T02:41:37.737996+0000","last_fullsized":"2024-08-16T02:41:37.737996+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:41:37.752497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:37.752497+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":658,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:36.790217+0000","last_change":"2024-08-16T02:41:36.813604+0000","last_active":"2024-08-16T02:41:36.790217+0000","last_peered":"2024-08-16T02:41:36.790217+0000","last_clean":"2024-08-16T02:41:36.790217+0000","last_became_active":"2024-08-16T02:36:30.539438+0000","last_became_peered":"2024-08-16T02:36:30.539438+0000","last_unstale":"2024-08-16T02:41:36.790217+0000","last_undegraded":"2024-08-16T02:41:36.790217+0000","last_fullsized":"2024-08-16T02:41:36.790217+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2024-08-16T02:41:36.813425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:36.813425+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:37.265843+0000","last_change":"2024-08-16T02:41:37.282460+0000","last_active":"2024-08-16T02:41:37.265843+0000","last_peered":"2024-08-16T02:41:37.265843+0000","last_clean":"2024-08-16T02:41:37.265843+0000","last_became_active":"2024-08-16T02:36:30.521005+0000","last_became_peered":"2024-08-16T02:36:30.521005+0000","last_unstale":"2024-08-16T02:41:37.265843+0000","last_undegraded":"2024-08-16T02:41:37.265843+0000","last_fullsized":"2024-08-16T02:41:37.265843+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:41:37.282288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:37.282288+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:36.790215+0000","last_change":"2024-08-16T02:41:36.807178+0000","last_active":"2024-08-16T02:41:36.790215+0000","last_peered":"2024-08-16T02:41:36.790215+0000","last_clean":"2024-08-16T02:41:36.790215+0000","last_became_active":"2024-08-16T02:36:30.548723+0000","last_became_peered":"2024-08-16T02:36:30.548723+0000","last_unstale":"2024-08-16T02:41:36.790215+0000","last_undegraded":"2024-08-16T02:41:36.790215+0000","last_fullsized":"2024-08-16T02:41:36.790215+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:41:36.807008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:36.807008+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":657,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:37.267665+0000","last_change":"2024-08-16T02:41:37.291626+0000","last_active":"2024-08-16T02:41:37.267665+0000","last_peered":"2024-08-16T02:41:37.267665+0000","last_clean":"2024-08-16T02:41:37.267665+0000","last_became_active":"2024-08-16T02:36:30.528787+0000","last_became_peered":"2024-08-16T02:36:30.528787+0000","last_unstale":"2024-08-16T02:41:37.267665+0000","last_undegraded":"2024-08-16T02:41:37.267665+0000","last_fullsized":"2024-08-16T02:41:37.267665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'1","last_scrub_stamp":"2024-08-16T02:41:37.291291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:37.291291+0000","last_clean_scrub_stamp":"2024-08-16T02:36:29.440942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":728,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:41:38.210064+0000","last_change":"2024-08-16T02:41:38.243152+0000","last_active":"2024-08-16T02:41:38.210064+0000","last_peered":"2024-08-16T02:41:38.210064+0000","last_clean":"2024-08-16T02:41:38.210064+0000","last_became_active":"2024-08-16T02:36:28.489045+0000","last_became_peered":"2024-08-16T02:36:28.489045+0000","last_unstale":"2024-08-16T02:41:38.210064+0000","last_undegraded":"2024-08-16T02:41:38.210064+0000","last_fullsized":"2024-08-16T02:41:38.210064+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'52","last_scrub_stamp":"2024-08-16T02:41:38.242921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:41:38.242921+0000","last_clean_scrub_stamp":"2024-08-16T02:36:27.432441+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198300672,"internally_reserved":0,"allocated":13193216,"data_stored":13193216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3472,"kb_used_data":3472,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294256,"statfs":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2836,"kb_used_data":2836,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294892,"statfs":{"total":53552873472,"available":53549969408,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2828,"kb_used_data":2828,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294900,"statfs":{"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3748,"kb_used_data":3748,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293980,"statfs":{"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53549969408,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:41:58.883 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-16T02:41:58.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.983+0000 7fa8c9592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:58.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.991+0000 7fa8c9592640 1 Processor -- start 2024-08-16T02:41:58.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.992+0000 7fa8c9592640 1 -- start start 2024-08-16T02:41:58.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c9592640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fa8c40a6ea0 0x7fa8c409fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:58.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c9592640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c40a7570 con 0x7fa8c40a0370 2024-08-16T02:41:58.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c9592640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c406f380 con 0x7fa8c40a6ea0 2024-08-16T02:41:58.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c2ffd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fa8c40a6ea0 0x7fa8c409fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:58.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8ba7fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fa8c40a0370 0x7fa8c4135320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:57008/0 (socket says 172.21.15.133:57008) 2024-08-16T02:41:58.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8ba7fc640 1 -- 172.21.15.133:0/3234516956 learned_addr learned my addr 172.21.15.133:0/3234516956 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-08-16T02:41:58.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c2ffd640 1 -- 172.21.15.133:0/3234516956 >> v1:172.21.15.133:6789/0 conn(0x7fa8c40a0370 legacy=0x7fa8c4135320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:58.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c2ffd640 1 -- 172.21.15.133:0/3234516956 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c40a1630 con 0x7fa8c40a6ea0 2024-08-16T02:41:58.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c2ffd640 1 --2- 172.21.15.133:0/3234516956 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a6ea0 0x7fa8c409fe30 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714d6441daae7d8b server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:58.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c27fc640 1 -- 172.21.15.133:0/3234516956 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa8b4009d90 con 0x7fa8c40a6ea0 2024-08-16T02:41:58.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c27fc640 1 -- 172.21.15.133:0/3234516956 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa8b400e230 con 0x7fa8c40a6ea0 2024-08-16T02:41:58.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.993+0000 7fa8c27fc640 1 -- 172.21.15.133:0/3234516956 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa8b400e4d0 con 0x7fa8c40a6ea0 2024-08-16T02:41:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.994+0000 7fa8c9592640 1 -- 172.21.15.133:0/3234516956 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a6ea0 msgr2=0x7fa8c409fe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.994+0000 7fa8c9592640 1 --2- 172.21.15.133:0/3234516956 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a6ea0 0x7fa8c409fe30 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.994+0000 7fa8c9592640 1 -- 172.21.15.133:0/3234516956 shutdown_connections 2024-08-16T02:41:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.994+0000 7fa8c9592640 1 --2- 172.21.15.133:0/3234516956 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a6ea0 0x7fa8c409fe30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.994+0000 7fa8c9592640 1 -- 172.21.15.133:0/3234516956 >> 172.21.15.133:0/3234516956 conn(0x7fa8c406ce60 msgr2=0x7fa8c406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.994+0000 7fa8c9592640 1 -- 172.21.15.133:0/3234516956 shutdown_connections 2024-08-16T02:41:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.994+0000 7fa8c9592640 1 -- 172.21.15.133:0/3234516956 wait complete. 2024-08-16T02:41:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.994+0000 7fa8c9592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:41:59.000 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.996+0000 7fa8c9592640 1 Processor -- start 2024-08-16T02:41:59.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.996+0000 7fa8c9592640 1 -- start start 2024-08-16T02:41:59.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.997+0000 7fa8c9592640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a0370 0x7fa8c4122f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:59.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.997+0000 7fa8c9592640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c41248f0 con 0x7fa8c40a0370 2024-08-16T02:41:59.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.997+0000 7fa8c2ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a0370 0x7fa8c4122f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:59.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.997+0000 7fa8c2ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a0370 0x7fa8c4122f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:42448/0 (socket says 172.21.15.133:42448) 2024-08-16T02:41:59.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.997+0000 7fa8c2ffd640 1 -- 172.21.15.133:0/2946389592 learned_addr learned my addr 172.21.15.133:0/2946389592 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-08-16T02:41:59.001 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.997+0000 7fa8c2ffd640 1 -- 172.21.15.133:0/2946389592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c41241a0 con 0x7fa8c40a0370 2024-08-16T02:41:59.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.997+0000 7fa8c2ffd640 1 --2- 172.21.15.133:0/2946389592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a0370 0x7fa8c4122f60 crc :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:59.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.998+0000 7fa8bbfff640 1 -- 172.21.15.133:0/2946389592 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa8b4014540 con 0x7fa8c40a0370 2024-08-16T02:41:59.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.998+0000 7fa8bbfff640 1 -- 172.21.15.133:0/2946389592 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa8b40146a0 con 0x7fa8c40a0370 2024-08-16T02:41:59.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.998+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c41251a0 con 0x7fa8c40a0370 2024-08-16T02:41:59.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.998+0000 7fa8bbfff640 1 -- 172.21.15.133:0/2946389592 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa8b4014940 con 0x7fa8c40a0370 2024-08-16T02:41:59.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.998+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c4125920 con 0x7fa8c40a0370 2024-08-16T02:41:59.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.999+0000 7fa8bbfff640 1 -- 172.21.15.133:0/2946389592 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fa8b4014af0 con 0x7fa8c40a0370 2024-08-16T02:41:59.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.999+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c4073620 con 0x7fa8c40a0370 2024-08-16T02:41:59.003 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.999+0000 7fa8bbfff640 1 --2- 172.21.15.133:0/2946389592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fa8a403dd10 0x7fa8a4040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:41:59.003 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.999+0000 7fa8bbfff640 1 -- 172.21.15.133:0/2946389592 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7fa8b4019040 con 0x7fa8c40a0370 2024-08-16T02:41:59.003 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:58.999+0000 7fa8ba7fc640 1 --2- 172.21.15.133:0/2946389592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fa8a403dd10 0x7fa8a4040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:41:59.003 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.000+0000 7fa8ba7fc640 1 --2- 172.21.15.133:0/2946389592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fa8a403dd10 0x7fa8a4040220 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:41:59.004 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.004+0000 7fa8bbfff640 1 -- 172.21.15.133:0/2946389592 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fa8c4073620 con 0x7fa8c40a0370 2024-08-16T02:41:59.190 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.190+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa8c4073620 con 0x7fa8c40a0370 2024-08-16T02:41:59.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.195+0000 7fa8bbfff640 1 -- 172.21.15.133:0/2946389592 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) ==== 123+0+0 (crc 0 0 0) 0x7fa8c4073620 con 0x7fa8c40a0370 2024-08-16T02:41:59.198 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.196+0000 7fa8bbfff640 1 -- 172.21.15.133:0/2946389592 <== mon.0 v2:172.21.15.133:3300/0 8 ==== config(1 keys) ==== 35+0+0 (crc 0 0 0) 0x7fa8b4012390 con 0x7fa8c40a0370 2024-08-16T02:41:59.199 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fa8a403dd10 msgr2=0x7fa8a4040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:59.199 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 --2- 172.21.15.133:0/2946389592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fa8a403dd10 0x7fa8a4040220 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:59.199 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a0370 msgr2=0x7fa8c4122f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:41:59.199 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 --2- 172.21.15.133:0/2946389592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a0370 0x7fa8c4122f60 crc :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:59.199 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 shutdown_connections 2024-08-16T02:41:59.200 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 --2- 172.21.15.133:0/2946389592 >> [v2:172.21.15.137:6800/545649490,v1:172.21.15.137:6801/545649490] conn(0x7fa8a403dd10 0x7fa8a4040220 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:59.200 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 --2- 172.21.15.133:0/2946389592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8c40a0370 0x7fa8c4122f60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:41:59.200 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 >> 172.21.15.133:0/2946389592 conn(0x7fa8c406ce60 msgr2=0x7fa8c4071730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:41:59.200 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 shutdown_connections 2024-08-16T02:41:59.200 INFO:teuthology.orchestra.run.smithi133.stderr:2024-08-16T02:41:59.198+0000 7fa8c9592640 1 -- 172.21.15.133:0/2946389592 wait complete. 2024-08-16T02:41:59.212 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-16T02:41:59.212 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-16T02:41:59.212 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-16T02:41:59.213 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:41:59.294 INFO:tasks.ceph.osd.0:Stopped 2024-08-16T02:41:59.295 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-16T02:41:59.295 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:41:59.364 INFO:tasks.ceph.osd.1:Stopped 2024-08-16T02:41:59.365 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-16T02:41:59.365 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:41:59.435 INFO:tasks.ceph.osd.2:Stopped 2024-08-16T02:41:59.436 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-16T02:41:59.436 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:41:59.511 INFO:tasks.ceph.osd.3:Stopped 2024-08-16T02:41:59.511 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-16T02:41:59.511 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-16T02:41:59.511 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:41:59.542 INFO:tasks.ceph.mgr.x:Stopped 2024-08-16T02:41:59.542 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-16T02:41:59.542 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-16T02:41:59.542 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:41:59.553 INFO:tasks.ceph.mon.a:Stopped 2024-08-16T02:41:59.554 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-16T02:41:59.554 DEBUG:teuthology.orchestra.run.smithi133:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-08-16T02:41:59.622 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi133.front.sepia.ceph.com 2024-08-16T02:41:59.622 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-16T02:41:59.774 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi133.front.sepia.ceph.com 2024-08-16T02:41:59.774 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-16T02:41:59.887 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi137.front.sepia.ceph.com 2024-08-16T02:41:59.887 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-16T02:41:59.976 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi137.front.sepia.ceph.com 2024-08-16T02:41:59.977 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-16T02:42:00.039 INFO:tasks.ceph:Archiving mon data... 2024-08-16T02:42:00.040 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420/data/mon.a.tgz 2024-08-16T02:42:00.041 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2024-08-16T02:42:00.056 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.oZ7Wz5vt0Y 2024-08-16T02:42:00.057 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.oZ7Wz5vt0Y 2024-08-16T02:42:00.197 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.oZ7Wz5vt0Y 2024-08-16T02:42:00.281 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.oZ7Wz5vt0Y is 357KB 2024-08-16T02:42:00.349 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.oZ7Wz5vt0Y 2024-08-16T02:42:00.364 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-16T02:42:00.365 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-16T02:42:00.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-16T02:42:00.483 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-16T02:42:00.499 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-16T02:42:00.500 INFO:tasks.ceph:Archiving crash dumps... 2024-08-16T02:42:00.501 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420/remote/smithi133/crash 2024-08-16T02:42:00.502 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-16T02:42:00.548 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420/remote/smithi137/crash 2024-08-16T02:42:00.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-16T02:42:00.584 INFO:tasks.ceph:Compressing logs... 2024-08-16T02:42:00.585 DEBUG:teuthology.orchestra.run.smithi133:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-16T02:42:00.593 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-16T02:42:00.614 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31406.log 2024-08-16T02:42:00.615 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30705.log 2024-08-16T02:42:00.615 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33218.log 2024-08-16T02:42:00.635 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31127.log 2024-08-16T02:42:00.636 INFO:teuthology.orchestra.run.smithi133.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.31406.log.gz 2024-08-16T02:42:00.636 INFO:teuthology.orchestra.run.smithi133.stderr: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.30705.log.gz 2024-08-16T02:42:00.636 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30441.log 2024-08-16T02:42:00.636 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31127.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.33218.log.gz 2024-08-16T02:42:00.636 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33344.log 2024-08-16T02:42:00.636 INFO:teuthology.orchestra.run.smithi133.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.31127.log.gz 2024-08-16T02:42:00.636 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30441.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-16T02:42:00.636 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30986.log 2024-08-16T02:42:00.637 INFO:teuthology.orchestra.run.smithi133.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.30441.log.gz 2024-08-16T02:42:00.637 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.33344.log.gz 2024-08-16T02:42:00.637 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31710.log 2024-08-16T02:42:00.637 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31129.log 2024-08-16T02:42:00.637 INFO:teuthology.orchestra.run.smithi133.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.30986.log.gz 2024-08-16T02:42:00.637 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31710.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-16T02:42:00.637 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30762.log 2024-08-16T02:42:00.638 INFO:teuthology.orchestra.run.smithi133.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-16T02:42:00.638 INFO:teuthology.orchestra.run.smithi133.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.31129.log.gz 2024-08-16T02:42:00.638 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30851.log 2024-08-16T02:42:00.638 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33555.log 2024-08-16T02:42:00.638 INFO:teuthology.orchestra.run.smithi133.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.30762.log.gz 2024-08-16T02:42:00.638 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32181.log 2024-08-16T02:42:00.638 INFO:teuthology.orchestra.run.smithi133.stderr: 88.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-16T02:42:00.638 INFO:teuthology.orchestra.run.smithi133.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30851.log.gz 2024-08-16T02:42:00.639 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33595.log 2024-08-16T02:42:00.639 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33555.log: /var/log/ceph/ceph-client.admin.32181.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-16T02:42:00.639 INFO:teuthology.orchestra.run.smithi133.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.33555.log.gz 2024-08-16T02:42:00.639 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31490.log 2024-08-16T02:42:00.639 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.1.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.33595.log.gz 2024-08-16T02:42:00.639 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-16T02:42:00.639 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31448.log 2024-08-16T02:42:00.639 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31490.log: /var/log/ceph/ceph-osd.0.log: 86.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31490.log.gz 2024-08-16T02:42:00.640 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30676.log 2024-08-16T02:42:00.640 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31296.log 2024-08-16T02:42:00.640 INFO:teuthology.orchestra.run.smithi133.stderr: 85.9%/var/log/ceph/ceph-client.admin.30676.log: -- replaced with /var/log/ceph/ceph-client.admin.31448.log.gz 2024-08-16T02:42:00.640 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33302.log 2024-08-16T02:42:00.640 INFO:teuthology.orchestra.run.smithi133.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.30676.log.gz 2024-08-16T02:42:00.640 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31028.log 2024-08-16T02:42:00.640 INFO:teuthology.orchestra.run.smithi133.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.31296.log.gz 2024-08-16T02:42:00.640 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30818.log 2024-08-16T02:42:00.641 INFO:teuthology.orchestra.run.smithi133.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.33302.log.gz 2024-08-16T02:42:00.641 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31028.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-16T02:42:00.641 INFO:teuthology.orchestra.run.smithi133.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.31710.log.gz 86.1% 2024-08-16T02:42:00.641 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31028.log.gz 2024-08-16T02:42:00.641 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33260.log 2024-08-16T02:42:00.641 INFO:teuthology.orchestra.run.smithi133.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30818.log.gz 2024-08-16T02:42:00.641 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31298.log 2024-08-16T02:42:00.641 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31130.log 2024-08-16T02:42:00.642 INFO:teuthology.orchestra.run.smithi133.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.33260.log.gz 2024-08-16T02:42:00.642 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33470.log 2024-08-16T02:42:00.642 INFO:teuthology.orchestra.run.smithi133.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.31298.log.gz 2024-08-16T02:42:00.642 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33092.log 2024-08-16T02:42:00.642 INFO:teuthology.orchestra.run.smithi133.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.31130.log.gz 2024-08-16T02:42:00.642 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31128.log 2024-08-16T02:42:00.642 INFO:teuthology.orchestra.run.smithi133.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.33470.log.gz 2024-08-16T02:42:00.642 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31297.log 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.33092.log.gz 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31295.log 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.31128.log.gz 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33134.log 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.31297.log.gz 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.31295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33050.log 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.31295.log.gz 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30902.log 2024-08-16T02:42:00.643 INFO:teuthology.orchestra.run.smithi133.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.33134.log.gz 2024-08-16T02:42:00.644 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33050.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29655.log 2024-08-16T02:42:00.644 INFO:teuthology.orchestra.run.smithi133.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.33050.log.gz 2024-08-16T02:42:00.644 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33008.log 2024-08-16T02:42:00.644 INFO:teuthology.orchestra.run.smithi133.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.30902.log.gz 2024-08-16T02:42:00.644 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph.tmp-client.admin.29655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33176.log 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29655.log.gz 2024-08-16T02:42:00.644 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-08-16T02:42:00.644 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33386.log 2024-08-16T02:42:00.644 INFO:teuthology.orchestra.run.smithi133.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.33008.log.gz 2024-08-16T02:42:00.645 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33512.log 2024-08-16T02:42:00.645 INFO:teuthology.orchestra.run.smithi133.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.33176.log.gz 2024-08-16T02:42:00.645 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30520.log 2024-08-16T02:42:00.645 INFO:teuthology.orchestra.run.smithi133.stderr: 85.7%/var/log/ceph/ceph-client.admin.33512.log: -- replaced with /var/log/ceph/ceph-client.admin.33386.log.gz 2024-08-16T02:42:00.645 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30944.log 2024-08-16T02:42:00.645 INFO:teuthology.orchestra.run.smithi133.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.33512.log.gz 2024-08-16T02:42:00.645 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33428.log 2024-08-16T02:42:00.645 INFO:teuthology.orchestra.run.smithi133.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.30520.log.gz 2024-08-16T02:42:00.646 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.30944.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.30944.log.gz 2024-08-16T02:42:00.646 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.33428.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.33428.log.gz 2024-08-16T02:42:00.646 INFO:teuthology.orchestra.run.smithi133.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.32181.log.gz 2024-08-16T02:42:00.655 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29626.log 2024-08-16T02:42:00.657 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-16T02:42:00.657 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29506.log 2024-08-16T02:42:00.657 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29626.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-08-16T02:42:00.657 INFO:teuthology.orchestra.run.smithi137.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.29626.log.gz 2024-08-16T02:42:00.657 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29506.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-16T02:42:00.657 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.3.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.29506.log.gz 2024-08-16T02:42:00.657 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29546.log 2024-08-16T02:42:00.658 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29586.log 2024-08-16T02:42:00.658 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29546.log: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.29546.log.gz 2024-08-16T02:42:00.658 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29586.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.29586.log.gz 2024-08-16T02:42:00.694 INFO:teuthology.orchestra.run.smithi137.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-16T02:42:00.878 INFO:teuthology.orchestra.run.smithi133.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-16T02:42:01.570 INFO:teuthology.orchestra.run.smithi137.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-16T02:42:01.593 INFO:teuthology.orchestra.run.smithi133.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-16T02:42:01.594 INFO:teuthology.orchestra.run.smithi137.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-08-16T02:42:01.595 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-08-16T02:42:01.595 INFO:teuthology.orchestra.run.smithi137.stderr:real 0m0.950s 2024-08-16T02:42:01.596 INFO:teuthology.orchestra.run.smithi137.stderr:user 0m1.818s 2024-08-16T02:42:01.596 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m0.095s 2024-08-16T02:42:01.740 INFO:teuthology.orchestra.run.smithi133.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-16T02:42:01.741 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-08-16T02:42:01.742 INFO:teuthology.orchestra.run.smithi133.stderr:real 0m1.137s 2024-08-16T02:42:01.742 INFO:teuthology.orchestra.run.smithi133.stderr:user 0m2.307s 2024-08-16T02:42:01.742 INFO:teuthology.orchestra.run.smithi133.stderr:sys 0m0.119s 2024-08-16T02:42:01.742 INFO:tasks.ceph:Archiving logs... 2024-08-16T02:42:01.743 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420/remote/smithi133/log 2024-08-16T02:42:01.744 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-16T02:42:02.190 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420/remote/smithi137/log 2024-08-16T02:42:02.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-16T02:42:02.507 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-16T02:42:02.516 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-16T02:42:02.516 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-16T02:42:02.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-16T02:42:02.602 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-16T02:42:02.664 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-16T02:42:02.664 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:42:02.667 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:42:02.679 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:42:02.682 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:42:02.683 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:42:02.683 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2024-08-16T02:42:02.684 INFO:teuthology.orchestra.run.smithi133.stdout:^* hv01.front.sepia.ceph.com 3 6 377 7 -24us[ -18us] +/- 40ms 2024-08-16T02:42:02.684 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 -579us[ -579us] +/- 77ms 2024-08-16T02:42:02.684 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +633us[ +633us] +/- 76ms 2024-08-16T02:42:02.684 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:42:02.686 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:42:02.686 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-08-16T02:42:02.686 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv01.front.sepia.ceph.com 3 6 377 7 -18us[ -15us] +/- 40ms 2024-08-16T02:42:02.686 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 -569us[ -569us] +/- 77ms 2024-08-16T02:42:02.686 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +636us[ +636us] +/- 76ms 2024-08-16T02:42:02.686 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:42:02.687 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-16T02:42:02.696 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-16T02:42:02.696 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-16T02:42:02.704 DEBUG:teuthology.orchestra.run.smithi133:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-16T02:42:02.785 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-16T02:42:02.846 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:42:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:02.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:42:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:02.905 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-16T02:42:02.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-16T02:42:02.922 INFO:teuthology.task.internal:Duration was 801.127216 seconds 2024-08-16T02:42:02.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-16T02:42:02.931 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-16T02:42:02.931 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-16T02:42:02.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-16T02:42:02.969 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:42:02.984 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:42:03.297 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-16T02:42:03.297 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2024-08-16T02:42:03.298 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-16T02:42:03.318 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-08-16T02:42:03.318 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-16T02:42:03.361 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-16T02:42:03.362 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-16T02:42:03.365 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-16T02:42:03.427 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-16T02:42:03.427 DEBUG:teuthology.orchestra.run.smithi133:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:42:03.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:42:03.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-16T02:42:03.575 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-16T02:42:03.575 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-16T02:42:03.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-16T02:42:03.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-16T02:42:03.637 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:03.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:03.665 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2024-08-16T02:42:03.688 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-08-16T02:42:03.728 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:03.742 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi133.front.sepia.ceph.com, flagging run as failed 2024-08-16T02:42:03.743 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:03.782 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-16T02:42:03.792 INFO:teuthology.task.internal:Transferring archived files... 2024-08-16T02:42:03.793 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420/remote/smithi133 2024-08-16T02:42:03.793 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-16T02:42:10.961 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-08-16T02:42:11.236 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857420/remote/smithi133/coredump/1723776092.32181.core.gz: gzip compressed data, was "1723776092.32181.core", last modified: Fri Aug 16 02:42:03 2024, max compression, original size modulo 2^32 139845632 2024-08-16T02:42:11.236 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-08-16T02:42:11.912 INFO:teuthology.task.internal: core looks like: /tmp/tmpfmu95234: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados_api_aio_pp', real uid: 1000, effective uid: 1000, real gid: 1316, effective gid: 1316, execfn: '/usr/bin/ceph_test_rados_api_aio_pp', platform: 'x86_64' 2024-08-16T02:42:11.926 INFO:teuthology.task.internal: dump_program: ceph_test_rados_api_aio_pp 2024-08-16T02:42:11.926 DEBUG:teuthology.orchestra.run.smithi133:> which ceph_test_rados_api_aio_pp 2024-08-16T02:42:11.956 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph_test_rados_api_aio_pp 2024-08-16T02:42:12.018 DEBUG:teuthology.orchestra.remote:smithi133:/usr/bin/ceph_test_rados_api_aio_pp is 1MB 2024-08-16T02:42:12.126 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2024-08-16T02:42:12.144 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.JjyLkStAEY 2024-08-16T02:42:12.145 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.JjyLkStAEY 2024-08-16T02:42:12.210 INFO:teuthology.orchestra.run.smithi133.stderr:cp: cannot stat '/usr/bin/ceph_test_rados_api_aio_pp.debug': No such file or directory 2024-08-16T02:42:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:12.212 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 363, in fetch_binaries_for_coredumps remote.get_file(debug_path, coredump_path) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 622, in get_file self.run(args=args) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.JjyLkStAEY' 2024-08-16T02:42:12.221 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-16T02:42:12.231 INFO:teuthology.task.internal:Not uploading archives. 2024-08-16T02:42:12.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-16T02:42:12.260 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-16T02:42:12.260 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-16T02:42:12.265 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-16T02:42:12.280 INFO:teuthology.orchestra.run.smithi133.stdout: 265874 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest 2024-08-16T02:42:12.280 INFO:teuthology.orchestra.run.smithi133.stdout: 405787 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:36 /home/ubuntu/cephtest/mnt.0 2024-08-16T02:42:12.280 INFO:teuthology.orchestra.run.smithi133.stdout: 405788 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:37 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-16T02:42:12.280 INFO:teuthology.orchestra.run.smithi133.stdout: 420599 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:37 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-16T02:42:12.280 INFO:teuthology.orchestra.run.smithi133.stdout: 265892 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive 2024-08-16T02:42:12.280 INFO:teuthology.orchestra.run.smithi133.stdout: 405786 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/audit 2024-08-16T02:42:12.280 INFO:teuthology.orchestra.run.smithi133.stdout: 405790 124 -rw------- 1 ubuntu ubuntu 124531 Aug 16 02:42 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-08-16T02:42:12.281 INFO:teuthology.orchestra.run.smithi133.stdout: 265937 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:42:12.281 INFO:teuthology.orchestra.run.smithi133.stdout: 262209 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 16 02:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-08-16T02:42:12.281 INFO:teuthology.orchestra.run.smithi133.stdout: 265418 192 -rw-r--r-- 1 ubuntu ubuntu 193641 Aug 16 02:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:42:12.281 INFO:teuthology.orchestra.run.smithi133.stdout: 265413 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 16 02:28 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-08-16T02:42:12.281 INFO:teuthology.orchestra.run.smithi133.stdout: 265904 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:41 /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:12.281 INFO:teuthology.orchestra.run.smithi133.stdout: 265458 87688 -rw------- 1 ubuntu ubuntu 139845632 Aug 16 02:41 /home/ubuntu/cephtest/archive/coredump/1723776092.32181.core 2024-08-16T02:42:12.281 INFO:teuthology.orchestra.run.smithi133.stdout: 420600 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:37 /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-16T02:42:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:12.282 INFO:teuthology.orchestra.run.smithi133.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-16T02:42:12.283 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 363, in fetch_binaries_for_coredumps remote.get_file(debug_path, coredump_path) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 622, in get_file self.run(args=args) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.JjyLkStAEY' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-16T02:42:12.293 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-16T02:42:12.303 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-16T02:42:12.312 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi133 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.JjyLkStAEY' 2024-08-16T02:42:12.316 INFO:teuthology.run:Summary data: description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests} duration: 801.1272158622742 failure_reason: 'Command failed (workunit test rados/test.sh) on smithi133 with status 134: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson''' flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f8888b9dc1fa43dfb03b15daacc48ff5 status: fail success: false 2024-08-16T02:42:12.317 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-16T02:42:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: 265306 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest 2024-08-16T02:42:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: 265307 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive 2024-08-16T02:42:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: 405786 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/audit 2024-08-16T02:42:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: 406092 116 -rw------- 1 ubuntu ubuntu 117595 Aug 16 02:42 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-08-16T02:42:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: 265311 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:42:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: 262209 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 16 02:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-08-16T02:42:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: 265312 188 -rw-r--r-- 1 ubuntu ubuntu 191458 Aug 16 02:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:42:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: 265313 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 16 02:28 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-08-16T02:42:12.325 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-16T02:42:12.402 INFO:teuthology.run:FAIL